Title of issue was: "Add leveldb to senic-os"
Now as we got leveldb python library installed and we started implementation/using it. We have multiple applications/process accessing DB and we ran into concurrency issues, we tried couple of things, like making sure every access opens DB connection and closes it but it didn't pass multiprocessing test,
import unittest
import os
import random
import string
from multiprocessing import Process
class TestDevicesDB(unittest.TestCase):
def setUp(self):
self.db = DbAPIWrapper('/tmp/test_db.db')
self.value = ''.join(random.choice(string.ascii_letters + string.digits) for _ in range(10))
def test_multipe_instance_access_to_db(self):
self.db.write("key", self.value)
self.assertEqual(self.db.read("key"), self.value)
p = Process(target=update_db)
p.start()
p.join()
self.assertEqual(self.db.read("key"), "new value")
self.db.delete("key")
self.assertEqual(self.db.read("key"), '')
p = Process(target=update_db)
p.start()
p.join()
self.assertEqual(self.db.read("key"), "new value")
leveldb documentation mentions that DB can be opened by only one process at a time and writing wrapper to confirm this condition(locks, semaphores) with python seemed a bit of too much work. We got a DB and its bindings setup on OS, but weren't able to use it. Back to square one.
This time, we thought of confirming that Berkeley DB indeed supports multiple process accessing DB. For Raspbian, there is package available for `berkeledb` and its python bindings, `bsddb3`. I installed them on a raspberry Pi, confirmed that above tests work and even accessing with multiple python instance accessing same DB and reading/writing values. Once this was confirmed, we knew that this DB will work for our requirements so we again resumed on getting the dependencies sorted out on Senic-OS.
Title of issue: "setup BerkeleyDB and bsddb3 on senic-os"
First thing to sort out on this task was how to get `berkeledb` binary tools on senic-os. Library was there, `libdb.so` but binary tools were not there. Some email threads mentioned about installing `db-bin` but running `bitbake db-bin` threw error that, `nothing provides db-bin`. I again reached out on IRC channel there good folks pointed me to this section of recipe mentioning that binaries would be directly installed on os if I included `db-bin` in list of packages. First step sorted out \o/
Second thing, though there was `libdb.so` file, somehow `bsddb3` recipe was not able to locate it. After some better insights on recipes from the work done on `leveldb`, I again looked at the initial recipe I had put together. I had to figure out how to pass extra arguments to `setup` tools of python, giving it location of folders where it can find `libdb.so`. This was a right question to ask and search, bitbake documentation and google helped and finally with following recipe, I was able to get `bsddb3` installed on senic-os:
SUMMARY = "pybsddb is the Python binding for the Oracle Berkeley DB"
HOMEPAGE = "https://www.jcea.es/programacion/pybsddb.htm"
SECTION = "devel/python"
LICENSE = "BSD-3-Clause"
LIC_FILES_CHKSUM = "file://LICENSE.txt;md5=b3c8796b7e1f8eda0aef7e242e18ed43"
SRC_URI[sha256sum] = "42d621f4037425afcb16b67d5600c4556271a071a9a7f7f2c2b1ba65bc582d05"
inherit pypi setuptools3
PYPI_PACKAGE = "bsddb3"
DEPENDS = "db \
python3-core \
"
DISTUTILS_BUILD_ARGS = "--berkeley-db=${STAGING_EXECPREFIXDIR}"
DISTUTILS_INSTALL_ARGS = "--berkeley-db=${STAGING_EXECPREFIXDIR}"
RDEPENDS_${PN} = "db \
python3-core \
"
We baked Senic-OS with these libraries and dependencies and ran the multip-processing tests and confirmed that indeed we were able to access DB from our multiple applications. And with this, the task got marked done and new task opened up of migrating applications to use this DB wrapper instead of using config files.