OpenPandora: Building your own ROOTFS image (Part 1).

Updated: 7th June to expand the setup scripts section and tell people to get them from GIT.

This is the 1st of 2 articles I will be publishing on getting started with building the Linux distribution installed on the OpenPandora, from source code, from scratch.

Note: This is not a guide to using OpenEmbedded or bitbake, or writing package recipes. It’s just a guide to getting an OpenPandora build setup running.

This 1st article will cover the basic setup of OpenEmbedded environment and the building of our Ångström distribution derived ROOTFS, the 2nd will detail how you can run the resulting images on your OpenPandora from SD cards, or if you really want to, the NAND.

I also plan to write a separate related article will show how you can setup toolchains and SDK’s to help you develop applications for (and natively on) the OpenPandora using some of the strengths of OpenEmbedded to make this easier.

As I write these posts in my own time I can’t promise to do anything quickly. Nor can I make any promises about how accurate they may be ;). As always, feedback is welcome.

Most application developers/porters will only be interested in the related article on setting up SDK’s and toolchains, these 2 are really for the hardened Linux hackers who want to mess about with the ROOTFS/distribution development.

Health (and sanity) warning: Building the entire ROOTFS from scratch is pretty technical in nature, if you have no desire to rebuild the ROOTFS or hack with the inner workings of you’re OpenPandora this is probably not for you.

You can’t really damage anything on your OpenPandora but if your not familiar with compiling your own apps, kernels etc. and fixing things when they don’t work this will present a VERY steep learning curve. You also have the potential to make a mess of your build host if you are not careful with the setup.

I’ll focus showing how you can build a ‘one <> one’ version of the 1st official release (GIT tag: “Release-2010-05/1”). 

Once you have that going feel free to build the tips of the metadata GIT’s if you want to work with the latest and greatest or get stuck in modifying the metadata to suit whatever purpose you may have in mind.


Contents:

Closed source prerequisites
Host OS for building the ROOTFS
Important GIT repositories
Setting up your environment for OpenEmbedded
Building the OpenPandora ROOTFS images using OpenEmbedded


Closed source prerequisites:

Before we start you will need to collect up the few closed source components used in the ROOTFS, thankfully Texas Instruments and Imagination Technologies at least make getting the closed source binaries and SDK for the PowerVR SGX quite easy these days.

PowerVR SGX Binaries and SDK:

You will need to download version 3.01.00.02 of the Ti Graphics SDK (OMAP35x_Graphics_SDK_setuplinux_3_01_00_02.bin) from http://software-dl.ti.com/dsps/forms/export.html?prod_no=/OMAP35x_Graphics_SDK_setuplinux_3_01_00_02.bin

You will have to create a Ti account to download the software and agree to a license/export agreement but once you have done this Ti normally instantly grant access to the download. After you download this store it in a safe place, you will need to drop it into a folder in the build system later on.

The other closed source components are the firmware that is run on the WL1251 wireless chip and Bluetooth chip (the drivers themselves are open source and in mainline Linux). However our build system will automatically downloads these closed source firmware binary firmwares and places them into the appropriate place in the ROOTFS.


Host OS for building the ROOTFS:

Note: all my OpenEmbedded build boxes are 32bit Ubuntu Linux 10.04 servers with PAE enabled kernels, building on most other Linux distributions should be (and is) simple enough however some small adjustments may need to be made to suit your distribution. The OpenEmbedded and Ångström documentation can help with this.

You will need to install a number of packages onto your host PC before you start building OpenEmbedded images.

Try the command below, this should get most of the major prerequisite packages installed that you will need to start building using OpenEmbedded on a Ubuntu/Debian derived distribution.

sudo apt-get install ccache sed wget cvs subversion git-core coreutils unzip texi2html texinfo libsdl1.2-dev docbook-utils gawk python-pysqlite2 diffstat help2man libxml2-utils xmlto python-psyco

Note: The OpenEmbedded Required Software page also details this very well and is a must read.

One last thing to consider if your using a Ubuntu derivative is that they use dash as the default shell not bash.

This can cause a few issues with some of the packages we want to try and build, it’s not a problem with the build system per-say but issues with the individual packages assuming bash hacks in whatever /bin/sh is on the system.

Still, the fix is pretty easy, just set your default shell to bash using the command below.

sudo dpkg-reconfigure dash


Important GIT repositories (GITWeb links):

All the metadata (and custom source code etc.) is held on a publically accessible GIT server (git.openpandora.org).

For the purposes of building a working ROOTFS the following GIT repositories are of interest.

Important remote branches are pandora-27-omap1 (Our ‘default’ kernel as of now) and linux-omap (synced with top of the mainline Linux-OMAP tree).

The important remote branch is master. This is our OpenEmbedded overlay (i.e. custom OpenPandora specific stuff we lay on top of OpenEmbedded/Angstrom to build our images).

The important remote branch is op.openembedded.dev, this is a snap shot from mainline OpenEmbedded with our queued patches stuffed on top. This serves as our feed tree for getting stuff into mainline OpenEmbedded/Angstrom (with a few clearly marked exceptions, look for checkins with HACK: in the commit message, these won’t be going to mainline ;)).

Should you with to manually clone any of the OpenPandora GIT repositories for whatever reason you can by issuing.

git clone git://git.openpandora.org/repository.name.git destination.folder

But if you just plan to build an image you have no need to do this as the setup script will take care of it for you.


Setting up your build environment for OpenEmbedded:

Using OpenEmbedded to build images is actually a pretty simple affair if done right however often steps are missed or done in the wrong order leading to all manner of spurious issues and resulting pain.

Security Warning: Never build your images as the ‘root’ user, there is no need to use a root account and doing so is a really bad idea, just make sure your normal user has control of the folder containing your OpenEmbedded environment.

To make the process as simple as possible I have knocked up a few simple support scripts that can just be installed to an appropriate place and run, hopefully setting up the environment correctly as they go.

These scripts are pretty crude and lack robust error checking and such so should be run with a note of caution. The idea is to improve them as we go.

Note: You will need LOTS of space to undertake a complete build of the ROOTFS, my typical build folders are often in excess of 100Gbyte :o. Make sure you have the space before you start ;).

1st off you need to pick somewhere on your system to construct the OpenEmbedded environment, this should be a path with no symlinks above it, a rebound mount point is fine however.

The user you want to use for builds will need control of this folder.

Open a terminal and change into the above folder that you wish to become the top level for your OpenEmbedded environment.

Get a copy of the setup scripts by running…

git clone git://git.openpandora.org/pandora-oe-environment.git .

Once you have the files checked out run the environment script (you must run it with . ./op-env-oe.sh so it goes into the calling shells environment).

. ./op-env-oe.sh

Now you have all the environment variables setup you can run the script that initialises the environment (creates folders, checks out the GIT’s etc.).

./initial-setup.sh

This will take a while to run as it creates directories, clones the necessary GIT trees and populates the environment. Now is a good time for the 1st coffee ;).

The next step is to switch from the tip of our GIT trees to the “Release-2010-05/1” tag.

This will ensure you are working with exactly the same metadata as the 1st release.

You can skip this step if you want to work with the latest and greatest code.

./use-release-2010-05-1.sh

Note: If you want to you can later switch back to the tip of the GIT trees (development metadata) using ./use-tip.sh.

Be careful if you have uncommitted changes in your local GIT tree as the scripts will try and ‘git stash’ them but I suspect that may not be what you want.

Now your environment is all setup (assuming the scripts worked ok ;)) your build folder should look like the breakdown below, more folders will be created as soon as you run a build (folders for sources, package staging and temp data for example).

top of build area \

  bitbake \ < Bitbake lives here.

  build \

     conf \

        local.conf < This is the file that tells OpenEmbedded about our setup.

  metadata \

     openpandora.oe.git \    < Checkout of master from openpandora.oe.git

     openembedded.git \     < Checkout of op.openembedded.dev from

                                               openembedded.git

     user.collection \     < Area for any user created package recipes.

  tmp \ < The temp area that all the building is done under.

The next step is to copy in the copy the PowerVR SDK you downloaded earlier into the right place in the build folder.

You need to copy OMAP35x_Graphics_SDK_setuplinux_3_01_00_02.bin into metadata/openembedded.git/recipes/powervr-drivers (so it ends up in the same folder as libgles-omap3_3.01.00.02.bb).

The command below should do it if you run it from the folder you downloaded the SDK into.

cp ./OMAP35x_Graphics_SDK_setuplinux_3_01_00_02.bin ${OEBRANCH}/recipes/powervr-drivers

Now you want to do a simple test to check everything is working as expected.

From the top of your build directory run…

bitbake nano

After a fair period of time (it will download and build a version of GCC to target the OpenPandora GCC so it can then use that GCC to build NANO ;)) you should have a successful build of the NANO text editing package.

Note: You can check this by looking in tmp/angstrom.5/deploy/glibc/ipk/armv7a to see the NANO ipk package (nano_2.0.9-r0.5_armv7a.ipk).

The ‘angstrom.5’ part of the path comes from the concatenation of angstrom and it’s internal distribution revision (currently 5).

Your now ready to move on to building something a little more complex…


Building the OpenPandora ROOTFS images using OpenEmbedded.

Now you have a suitable environment setup you can move on to building your 1st image.

Every time you open a new shell/terminal you will need to call the environment setup script to create the necessary environment variables, the rest of the setup is already done.

. ./op-env-oe.sh

You have the full Angstrom and OpenEmbedded metadata at your disposal but for the purposes in hand there are a 2 main types of OpenPandora image stored in our OpenEmbedded overlay and 2 additional ones showcasing other Desktop environments.

You can see if a package is in a given image by looking at the corresponding recipe task file (found in metadata/openpandora.oe.git/recipes/tasks).

They are as follows (image name – task used).

  • pandora-core-image – task-pandora-core

This image contains all of the core packages for the OpenPandora but lacks any GUI/X Server etc. – It is lightweight (reasonably) and obviously CLI. Ideal if your working on hardware drivers etc. and don’t need a GUI layered on top. It also features things like libPND baked in.

  • pandora-xfce-image – task-pandora-xfce

This is the default OpenPandora image that ships on the NAND. It contains everything in the core image above and adds XOrg, Xfce and MiniMenu as the GUI elements. It also contains a myriad of GUI applications, scripts and the like as you would expect.

  • pandora-desktop-image – task-pandora-desktop

This is an Enlightenment (E17) based variant of the Xfce image above, it’s not had much love for a little while but should broadly be comparable to the Xfce image if you swapped out Xfce for E17 ;). I am hoping this image will get a little love ‘post release’ to bring it in line with the Xfce image.

  • pandora-gui-image – task-pandora-gui

This is a Matchbox based variant of the Xfce image above, it’s not had much love for a little while but should broadly be comparable to the Xfce image if you swapped out Xfce for Matchbox ;). This image feels very ‘PDA’ like and less ‘pocket Linux system’.

pandora-desktop-image and pandora-gui-image are not really images intended for end users but rather then remove them from the tree they have been left so interested parities can mess about with them.

Remember that building any of these images from scratch is going to take several hours even on the fastest of boxes.

Building the OpenPandora release image is now is as simple as running

bitbake pandora-xfce-image

from the top level folder of your newly created build tree. Make sure you have run the environment setup script first or it wont find your metadata.

If all goes well just go away and have a coffee (or 20) and check every now and then to see how things are going.

It’s going to take a good few hours to build everything and there may be the odd snags along the way, if you run into troubles drop me a line and I’ll see if I can help.

Note: Once the build finishes you can check the output in tmp/angstrom.5/deploy/glibc/images/omap3-pandora, with a little luck you should have a version of the pandora-xfce-image image ready to go. A tar.bz2 (ideal for extracting to an EXT 2/3 SD card) and a .ubifs.img (ready to flash to the NAND).

I’ll cover what you should do to get your newly generated ROOTFS running on the OpenPandora in the next article.


Known Issues

OpenEmbedded is a complex beast at the best of times and can often throw up issues when you least expect things. As long as your building on recent, fairly standard, Debian based (so Ubuntu et. all) type distributions you should not run into much trouble.

I can’t speak for distributions like SuSE, Arch, Fedora etc. as I don’t use them.

One known issue that seems to crop up (especially with the release-2010-05-1 metadata, I have a fix lined up for the trunk) is that the build will fail complaining about an issue with the ‘encodings’ package.

Should you see that issue the fix is very simple.

bitbake -cclean encodings

Then you just need to rerun whatever command you had run before (don’t worry, it won’t run all the steps again) and everything should be just fine.

The encodings issue is caused by mixed up dependences and cleaning it out and letting OpenEmbedded rebuild it sorts out those issues.

If you notice other issues do please let me know (or better still send patches).

Regards,

John

20 Comments



  1. Sinoth, I had problem with bluez4 too, your tip works for me, thanks.


  2. Had to install the two packages “rarian-compat” and “gnome-doc-utils” to get zenity to compile.

    Sorry about all the comments, I am posting fixes as I go. I am starting from a completely stock Ubuntu 10.04 server, so it’s to be expected I’m missing a few packages 🙂


  3. I suggest adding the “desktop-file-utils” package to the list of recommended packages. I was getting some “desktop-file-validate: command not found” warnings which are probably harmless, but disappear when the above package is installed 🙂


  4. I was assuming NETLINK was ‘libnfnetlink’, but it is actually looking for ‘libnl’ 🙂 What you need to do is “bitbake libnl” and then “bitbake bluez4”. The process should continue after that.


  5. Ok, stumped for the moment. I can manually build libnfnetlink, and I’ve added “libnfnetlink” to the DEPENDS line of the bluez .inc, but still the bluez config fails on the ‘checking for NETLINK’ part.

    Is there something else that has to be done for the bluez config to see libnfnetlink?


  6. I got an error on the first try through on package bluez4-4.62-r6.0 for the following reason:

    | checking for NETLINK… no
    | configure: error: Netlink library is required

    I guess Netlink isn’t getting compiled before bluez tries to compile? Will investigate, I guess by poking the recipies…


  7. Sorry for posting so many comments.
    Unfortunately I’ve got another error while compiling gcc-cross-initial-4.3.3-r11.1. The assembler complains about an unrecognized option ‘-Qy’.

    http://pastebin.com/CryALu96


  8. Ok, so I figured this one out myself. For some reason the macro S_ISDIR was used in src/path.c of the m4 package, but the header file defining that macro (sys/stat.h) hasn’t been included. Adding

    #include

    to the beginning of that file and repackaging the fixed m4 solved this issue. Let’s see how far I get this time. 😉


  9. Sure thing. The machine I’m trying to build the environment on is a AMD64 dual-core machine with 8 GB of RAM running 64 bits ArchLinux.

    Linux computer 2.6.33-ARCH #1 SMP PREEMPT Sun May 2 10:40:03 CEST 2010 x86_64 AMD Athlon(tm) Dual Core Processor 4850e AuthenticAMD GNU/Linux

    The only log file I could find was ./tmp/angstrom.5/work/x86_64-linux/m4-native-1.4.14-r0.0/m4-1.4.14/config.log which is the log output of the configure script run of M4.

    http://pastebin.com/1zuhahjh

    Thanks for your help so far.:)


  10. Thanks for fixing that, but unfortunately it still doesn’t work. Before you’ve added the local.conf I had tried to create my own local.conf, which worked somewhat, but finally failed when trying to run “bitbake nano”. Even with your local.conf it failed with the very same error:

    ERROR: ‘/home/w/pandora/oe/metadata/openembedded.git/recipes/m4/m4-native_1.4.14.bb’ failed


    1. Care to paste up the error log somewhere? It’s a recipe failing to build (the native/host version of M4 in this case), so to get that far it looks like your environment is fine.

      The error log linked to from the end of the build will provide some insight into what is going wrong.
      You could also provide some helpful background like the distro your using, your setup etc.

      Regards,

      John


  11. When following your instructions, there is no build/conf/local.conf file. Without that file using bitbake immediately fails with a fatal error.


    1. Now fixed, I forgot to push the local.conf from my local GIT to the server, sorry about that.


  12. Thanks John for you reply.

    How about that:
    http://www.gp32x.com/board/index.php?/topic/53964-cross-compiling-is-bollocks-toolchains-are-bollocks-use-your-pandora-to-build-apps

    It is about compiling straight on the device, example application is using 3D (opkg install libgles-omap3-dev libgles-omap3-demos).

    So is it possible to develop 3D-applications without TI’s binary-blob? (and it is used only when compiling driver/full image?)

    Because on Nokia N900, developer can’t compile anything depending gles without signing their license: http://tablets-dev.nokia.com/eula/index.php


    1. Hi Aapo,

      Regarding the post you linked to, I actually helped Torpor get native development going (I have built code on the device numerous times during development) and I have an article on setting up development environments and toolchains in my drafts that I need to finish off. I’ll do that in the next few days.

      Anyway, I suspect you misunderstand the relationship between the closed source blob that talks to the PowerVR SGX hardware and OpenGLes development.

      We have permission to distribute the binary blobs with the OpenPandora hardware (and as I suggested, end users can also request the binaries and full SDK from Ti directly).

      We are also allowed to package up the development headers and the like to allow users to talk to the OpenGLes (and OpenVG) standard side of the binary driver. This is what you will find in packages like libgles-omap3-dev.

      The basic workflow, in very simple terms, is that you send OpenGLes/OpenVG commands to the binary driver and this then executes the appropriate commands on the actual hardware.

      We do not require users to sign up to any NDA/Agreements etc. to get packages such as libgles-omap3-dev (or the libgles-omap3-demos etc.). This is perfectly sufficient for undertaking OpenGLes development for the OpenPandora.

      If the user wishes to obtain the full SDK from Ti directly then they are more than welcome to do so using the link I provided but the relationship and agreement is then between the user and Ti at that point.

      The reason people are asked to request the binary drivers and full SDK from Ti to build the image is simple, that is closed source and we cannot redistribute the entire package, the recipes (i.e. scripts) that then take the package apart and build the libgles-omap3-* packages that are used in the image are part of the image build scripts and can be found in GIT.

      It is the individual resulting libgles-omap3-* packages that we have permission to distribute not the full package.

      In simple terms if all you want to do is OpenGLes/OpenVG development for the OpenPandora you do not need to sign any agreements to do so. If you want the full SDK and binaries from Ti complete with sample code, docs and examples you will need to request this directly from Ti.

      I hope this helps clarify the situation.

      Regards,

      John


  13. Thank you. This is instructions I have been looking for. Just got nano.ipk built and waiting powervr-drivers.

    But does this closed source driver mean: When TI decides to not provide this driver anymore, I can’t then make 3D development for my 100% open device? (Actually I do not have my device yet, but soon)


    1. Aapo,

      It has always been clear that the PowerVR drivers are the one aspect of the system that is not open source.

      However there are several efforts underway in the wider community to create open source drivers for the PowerVR SGX (and the Intel GMA500, a PowerVR SGX variant). Keeping polite, gentle, pressure on Texas Instruments and Imagination Technologies to be more open with the specs would not hurt in this regard.

      The shim from the closed source driver to the open source kernel is open source code (GPL) and should Ti ever cease support this could still be maintained to allow the closed source driver components to work on newer kernels. Ti are also committed to supporting the PowerVR SGX driver for the long term (read into that what you may).

      I hope that helps clarify things.

      Regards,

      John

Comments are closed.