ISP Snoopgate

I was looking at NMD website, for some reference and on the slideshow I noticed "Lorem Ipsum". Now for a design school website that itself is very damning. Further, on clicking on one of those sliders, a new tab was opening for me, taking me to a random site. This was worrying, as the site belonged to a government institute and is serving or doing something malicious. I reached out to Nandeep, to see if he was also seeing similar behaviour, he was not. I tried different browsers, with Ad blocks and the anomaly was gone. Nandeep joked that maybe I got pawned, which got me a bit worried. On one side I was feeling, nah, some random behavior, ghost in the shell, and on other side I was cursing myself for not being able to understand what just happened on my own system.

I tried to reproduce the behaviour again, disabled all adblockers, and opened up inspector and looked at the network requests. At the bottom of the requests, I noticed a few weird http requests being made to IP on port 3000. Here is a screenshot of request/network over http and https: comparitive.png

Some context here, Indian broadband service providers have often been caught with their hands stuck in cookie jars(Airtel, BSNL). And their official response has been a shrug and something on the lines of "This is a standard solution deployed by telcos globally, meant to improve customer experience and empower them to manage their usage". It is one thing reading such incidents happening to others and a totally different thing noticing it first hand. Though the MITM was working as it was explained in classes of cryptography-101, but experiencing it in person, on live system was giving me a head rush.

I looked at params the request was sending and it included fields like subscriberId, subscriberIP: params.png

At this point I reached out to punch bhai for comments and feedback. I shared screenshots of network requests and after comparing them to his network requests, he helped me zero in to the origin of these dubious requests. After scrolling through the requests, we finally came to request made for JQuery-1.6.1.min.js and the ISP was shorting the request and passing me another Javascript file which internally did the shady requests to it's own IP and also loaded original JS file. request_shorting.png

This was nuts. We tried couple of other sites, like and there too, BSNL was injecting script. jquery_site.png

I enabled https everywhere browser extension and with that in place the whole interaction with the site was clean. Given that in spite of previously reported incidents, ISPs don't care and keep behaving the way they like, we as consumers don't have much choice left. Unless, our parliament passes a privacy law as directed by Supreme Court, something on the lines of GDPR, and usable and powerful like RTI. There are already draft bills available, SaveOurPrivacy is backed by Internet freedom foundation, which had worked really hard to push TRAI for NetNeutrality in India. There is nice video compiled by people behind this draft: Let us keep pushing for such a law which empowers uss, the customers, to demand the ISPs to furnish data they are collecting, with whom they are sharing it and how much they are sharing and if they are invading our privacy, to follow a legal course of action.

Getting the act together, personally.

I started working on SocialMedia Feed project from last August. And time after time I had found myself in a slump, looking for motivation, head rush to finish a task. Try to watch movie, find some song-of-day, something, anything, and before I realize, its the next day already. They say, “Show up, show up, show up, and after a while the muse shows up, too” and personally while trying it, self motivation drags, in really bad way.

Last week I was trying to put together a POC for a possible paid work. Task was to create a chat bot around different columns of Excel sheet for users to be able to chat to and get analytical results in conversational manner. So instead of a technical person running a query raised by sales team, user can directly chat with a bot and get answer to a question like "How many tasks finished successfully today", tasks could be something like email campaign or nightly aggregation of data or results from a long running algorithm. Airtifical Intelligence Markup Language - AIML and its concepts are fairly popular to put together such a chat bot. Many of popular platforms like pandora bots,, help create such interface to "train" a bot. Like in case of the above message - How many tasks finished successfully today tasks, successful, today gives us the context on which column to run the query, what value to look for and duration over which we want to get the count.

I didn't have prior experience of using platform so I had planned to read up some docs and train the system to identify context, maybe a couple of them - ~2hrs of effort. I ended up starting to work on it only past 12 pm, browsing through random doc links, exploring sdk for examples, trying to find my way through setting up the pipeline. By evening 6, I had clocked around 3 hours and 30 minutes on this task, I got context in place, provided webhook to call third party API call so I used their github demo code and got result from the excel sheet which was shared, though very minimal work but still, I was able to sort out first level of unknowns.

It was frustrating, tiring, but I felt this pressure to finish it. Though the POC worked, I didn't get the work but what is really sad is the way I was able to stick to deadline when pressure of proving myself, convincing someone else was there. For SocialMedia Feed project, I have this task, on top of basic gensim topic words, implement algorithms from Termite and LDAvis paper to identify better topic representation and get this demo in place. I have reference code available(paper work is available on github), I know what I have to do, but still through last 3 days I haven't clocked single minute on this task. It will happen eventually, but I think idea is to get fired up personally, take on things, spend time on them and then mark them done, be professional, not just for others.

Shoutout to punchagan for his inputs on initial draft.

Service worker adventures

With SoFee major work is done in background using celery, polling twitter for latest status, extract the links, fetch their content and eventually the segregation of content would also be done this way. I was looking for a way to keep things updated on user side and concepts of Progressive web app were really appealing.

What does it do?

Browsers(google chrome, firefox et all) are becoming more capable as new web standards are rolling out, like having offline cache, push notifications, accessing hardware(physical web). With these features now HTML based websites can also work as an native app working on your phone(android, iPhone) or desktop.

Stumbling Block #1: Scope and caching

I am using Django and with it all static content(css, JS, fonts) gets served from /static. And for service workers, if we do that, its scope gets limited to /static, that is, it would be able to handle requests getting served under /static. This limits access to API calls I am making. I Looked around and indeed there was a stack-overflow discussion around the same issue. Its a hacky solution and I added on to it by passing on some get PARAMS which I can use in template rendering for caching user specific URLs.

Beyond this I had a few head scratchers while getting cache to work. I struggled quite a bit to short the fetch request and return cached response but it just won't work. I Kept on tweaking the code, experimenting things until I used Jake's trained-to-thrill demo as base to setup things from scratch and then build on top.

Stumbling Block #2: Push Notifications

Service worker provides access to background Push notification. In earlier releases, browsers would register for this service and return a unique Endpoint for subscription, a unique capability URL which is used by server to push notification to. While this endpoint provided by Firefox works out of box, for chromium and google chrome browser, it still returned an obsolete GCM based URL. Now google has started using Firebase SDK and GCM is no longer supported. Beyond this on service side PyFCM library worked just fine to push notifications and it works with firefox too.