In this two part posts, I am trying to document the process of deciding on a issue, and how it evolves while trying to resolve or "close" it.
After initial release of Nuimo Click, we looked at the pain points we had on our backend stack. Currently we use lot of config file across processes, some applications write to them, others read/need them to connect to smart home devices. We use threads which "watches" changes to these config file so applications update themselves. This overall is becoming pain and also leaving us in nondeterministic states, which resulted in bad UX and is hard to reproduce and debug. We had thought of using Database or better key-value storage system for one sprint and I tried to take a shot at it.
Title of issue: "Add DB to senic-os"
Avichal my colleague, had already done some benchmarking with options
available and I started with this reference point. There was initial
consensus on using popular BerkeleyDB, said library(libdb.so) was
already part of the SenicOS and we just wanted python3 library recipe
for it to start using it.
Title of issue: "Add BerkeleyDB to senic-os"
I started exploring on how to get the library installed but got stuck with following bug:
user@senic-hub-02c0008185841ef0:~# ls -la /usr/lib/libdb-5.so
lrwxrwxrwx 1 user user 12 May 15 13:44 /usr/lib/libdb-5.so -> libdb-5.3.so
user@senic-hub-02c0008185841ef0:~# pip3.5 install bsddb3
Collecting bsddb3
Downloading https://files.pythonhosted.org/packages/e9/fc/ebfbd4de236b493f9ece156f816c21df0ae87ccc22604c5f9b664efef1b9/bsddb3-6.2.6.tar.gz (239kB)
100% | | 245kB 447kB/s
Complete output from command python setup.py egg_info:
Can't find a local Berkeley DB installation.
(suggestion: try the --berkeley-db=/path/to/bsddb option)
----------------------------------------
Small background, Yocto is custom embedded linux distribution which can be tailored for any SoC to create a small footprint, lightweight distribution. For libraries and packages which needs to be shipped with the OS, we need to write recipes. They are small snippet of config file, which defines the components, source of the package, dependency, tools needed to compile the recipe etc. Most/many packages already have recipes for them in layer-index, where they can be searched and integrated out of the box, but at times, we need to write one.
I am familiar with and writing/figuring out how to put together a
recipe which should work but I needed some more understanding of
internals. While python package, bsddb3 is actively maintained,
Berkely DB itself was available for download behind oracle sign-in. I
wasn't able to get these dependencies sorted out hence I started
looking at alternatives.
leveldb was a good option, it is actively maintained, has a recipe available for its core package and its python library is also well maintained.
Title of issue: "Add leveldb to senic-os"
As I tried to put together recipe for leveldb, I got stuck on trying
to figure out how to make cython compile header files from the
library for the native platform we used. I reached out in IRC
channel(#oe on irc.ubuntu.com) and shared my recipe and doubt, folks
there helped me understand how to use, enable cython to compile for
native platform. Here is how the final recipe looked like:
DESCRIPTION = "Plyvel is a fast and feature-rich Python interface to LevelDB."
HOMEPAGE = "https://github.com/wbolster/plyvel/"
SECTION = "devel/python"
LICENSE = "BSD"
LIC_FILES_CHKSUM = "file://LICENSE.rst;md5=41e1eab908ef114f2d2409de6e9ea735"
DEPENDS = "leveldb \
python3-cython-native \
python3-setuptools-native \
"
RDEPENDS_${PN} = "leveldb"
# using setuptools3 fails with make command
# python3native is needed to compile things using python3.5m
inherit setuptools python3native
S = "${WORKDIR}/git"
SRC_URI = "git://github.com/wbolster/plyvel.git;tag=1.0.5 \
file://setup.patch \
"
PV = "git-${SRCPV}"
I needed to add a small patch to python package to get it compiled
with python3:
diff --git a/Makefile b/Makefile
index 2fec651..2c2300a 100644
--- a/Makefile
+++ b/Makefile
@@ -3,8 +3,8 @@
all: cython ext
cython:
- cython --version
- cython --cplus --fast-fail --annotate plyvel/_plyvel.pyx
+ cython3 --version
+ cython3 --cplus --fast-fail --annotate plyvel/_plyvel.pyx
ext: cython
python setup.py build_ext --inplace --force
diff --git a/setup.py b/setup.py
index 3a69cec..42883c6 100644
--- a/setup.py
+++ b/setup.py
@@ -1,7 +1,10 @@
from os.path import join, dirname
from setuptools import setup
+from distutils.command.build import build
from setuptools.extension import Extension
import platform
+from distutils.command.install import install as DistutilsInstall
+from subprocess import call
CURRENT_DIR = dirname(__file__)
@@ -14,6 +17,16 @@ def get_file_contents(filename):
return fp.read()
+class BuildCython(build):
+ def run(self):
+ cmd = 'make'
+ call(cmd)
+ build.run(self)
+ # do_pre_install_stuff()
+ # DistutilsInstall.run(self)
+ # do_post_install_stuff()
+
+
extra_compile_args = ['-Wall', '-g']
if platform.system() == 'Darwin':
extra_compile_args += ['-mmacosx-version-min=10.7', '-stdlib=libc++']
@@ -53,5 +66,8 @@ setup(
"Topic :: Database",
"Topic :: Database :: Database Engines/Servers",
"Topic :: Software Development :: Libraries :: Python Modules",
- ]
+ ],
+ cmdclass={
+ 'build':BuildCython
+ }
)