Using Generators to flatten a JSON doc in PHP

To flatten a JSON like this:

{
   "addresses" : [
     {
       "line1" : "123 Livingstone Rd",
       "id" : 23444555,
       "addressSummaryType": "Building",
       "subAddresses": {
         "Level1": [
           {
             "line1" : "Level 2",
             "id" : 266887373,
             "Level2": [
               {
                 "line1": "Suite 201",
                 "id" : 276888890
               }
             ]
           }
         ]
       }
     },
     {
       "addressSummaryType": "Building",
       "line1" : "124 Livingstone Rd",
       "id" : 2562672
     }
   ]
 }

to:

array(2) {
  ["id"]=>
  int(23444555)
  ["address"]=>
  string(18) "123 Livingstone Rd"
}
array(2) {
  ["id"]=>
  int(266887373)
  ["address"]=>
  string(26) "Level 2 123 Livingstone Rd"
}
array(2) {
  ["id"]=>
  int(276888890)
  ["address"]=>
  string(36) "Suite 201 Level 2 123 Livingstone Rd"
}
array(2) {
  ["id"]=>
  int(2562672)
  ["address"]=>
  string(18) "124 Livingstone Rd"
}

I used the following code:

foreach(flattenAddress($addressObj->addresses) as $flattenedAddress) {
      var_dump($flattenedAddress);
 }

 function flattenAddress($addresses, $baseAddress = '', $level = 0)
 {
     foreach ($addresses as $address) {
         $nextLevel = "Level" . ($level + 1);
         if (property_exists($address, "addressSummaryType")) { //root
             $baseAddress = $address->line1;
             yield ["id" => $address->id, "address" => $baseAddress];
             if (property_exists($address, "subAddresses")) {
                 yield from flattenAddress($address->subAddresses->{$nextLevel}, $baseAddress, ($level + 1));
                 continue; // recurse don't reset level
             }
         } else { //subaddresses
             $subAddress =  $address->line1 . " " . $baseAddress;
             yield ["id" => $address->id, "address" => $subAddress];
             if (property_exists($address, $nextLevel)) {
                 yield from flattenAddress($address->{$nextLevel}, $subAddress, ($level + 1));
                 continue; // recurse don't reset level
             }
         }
         $level = 0; //reset for next root
     }
 }


Add extra validation rules to Laravel ForgotPassword

I wanted to add a captcha to the Forgot Password form, so I ended up doing this to the default ForgotPasswordController:

use Validator;
use \Illuminate\Http\Request;
 
class ForgotPasswordController extends Controller {
 
protected function validateEmail(Request $request)
{
    Validator::make(
        $request->all(),
        [
            'email' => 'required|email',
            'captcha' => 'required|captcha', // extra param to be validated
        ]
   )->validate();
}

Only relevant bits are shown there.

ES6 support on iOS9

Recently while building a Cordova based app, few users complained that they were seeing blank pages or cryptic error messages. Someone mentioned they were on iOS9. Over the weekend, as I ran the app on a simulator, I noticed an error in Safari. And, that’s when I realized ES6 features and syntax is not fully supported in iOS9.  Please see here: ES6 Compatibility (check unstable platforms)

So be-aware of ES6 features compatibility (unless you can transpile/polyfill every bit of code)

Jenkins parallel pipeline

I was very excited to know about the ‘parallel’ feature in Jenkins Pipeline, but, there are many gotchas while making  use of the pipeline feature (many of which are documented here: Jenkins Pipeline Example). After trying and reading a few different solutions, following worked for me (notice in screenshot that the browser jobs run in parallel !)

pipeline {
  agent any
  stages {
    stage('Build') {
      steps {
        echo 'hello'
      }
    }
    stage('Test') {
      steps {
        script {
            def jobs = [:]
            def browsers = ["Chrome", "Firefox"]
            for (int i = 0; i < browsers.size(); i++) {
                def browser = browsers.get(i)
                def jobName = "printing $browser"
                jobs[jobName] = doJob(browser)
            }
            parallel jobs
        }
      }
    }
    stage('Deploy') {
      steps {
        echo 'deploying'
      }
    }
  }
}

def doJob(browser) {
    return {
        node {
            echo "testing in $browser"
        }
    }
}

React Native: Application is not registered

- This is either due to a require() error during initialisation or failure to call AppRegistry.registerComponent.

I had been struggling with this for some time and the answer was right there… I had to make sure I called AppRegistry.registerComponent with the right params/appname – I had renamed my app and forgot to update the class name and also the name passed to registerComponent.

And, one way to rename a project is by renaming in package.json and then running react-native upgrade.

High System CPU usage

After upgrading a webserver running Apache to Debian Jessie (from Wheezy), I noticed that the system CPU usage was higher.  Running an strace on one of the Apache processes was giving me very little info:


strace -c -p 10112
Process 10112 attached - interrupt to quit
^CProcess 10112 detached
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
95.71 28.549784 3172198 9 4 futex

I had previously tried to spread interrupts across cores and also limited network activity – as they are something one can do to work out why System CPU is high – but they didn’t bring down the CPU usage.

One culprit remained: Futex/Mutex locks. I changed the default Mutex [1] to file and magically the System CPU usage went down.

[1] Apache 2.4 Mutex doc

HHVM notes

  • Impressive throughput improvements (>100%) with the app that I am working on.
  • phpinfo() doesn’t output what you would expect.
  • xhprof output_dir doesn’t get read from ini files, need to set that up in the constructor of XHProfRuns_Default.
  • Set hhvm.server.thread_count to a high value (>=MaxRequestWorkers), otherwise a few slow MySQL queries could bring the server to halt, minimal doc here: HHVM server architecture (worker thread => hhvm.server.thread_count). Suggest to keep it higher while JITing is happening.
  • If using Newrelic, tough luck!
    Unofficial Newrelic HHVM extension uses XHProf internally, so cannot get any data out of your own XHProf usage.
    The extension above relies on agent SDK that has no support for MySQL slow traces.
    Very low MySQL time in transactions.
    Strange traces in transactions.
  • CGI differences (apache_getenv not available use $_SERVER, SCRIPT_NAME will not be the same as REQUEST_URI).
  • Use realpath in imageftbox, relative paths for fonts don’t work.
  • Use Apache 2.4 as it has FastCGI support.
  • hhvm.log.header = true to have datetime in hhvm log.
  • HHVM log will also contain slow sql.
  • .hhbc was getting very huge, turned out it was due to Smarty file caching being enabled (the cached files were themselves php files that HHVM was compiling).
  • .hhbc file is sqlite(3) file that one can query (that is how I worked out the above).
  • High timeout values in memcached was leading to very high System CPU usage.
  • @ wasn’t suppressing (this could be Newrelic related)
  • Friendly folks in the hhvm IRC channel (get link from HHVM homepage), need to be online during daytime in the US.

HTTPS and GA referral exclusion

After converting a site to be served over TLS, we noticed the drop off rates increasing on some pages. This was because: user was being redirected to a payment page on another domain that was also served over TLS. As, now both sites were served over TLS, Google Analytics was picking up the referrer (HTTPS -> HTTP would pass no referrer, HTTPS -> HTTPS does) and was counting the payment page/domain as a new source. Thus, counting the redirect as an exit. Fix was simple, I excluded the payment domain in the referral list.

Google Analytics referral exclusion

casperjs output to html

Documenting what I had to do.

Used XSLT from here:
nosetest xslt

Problems and Fixes:

  1. Firefox was inserting “transformiix” as the root element, this caused the DOCTYPE to be spit out. I fixed by adding: 
    doctype-public="-//W3C//DTD HTML 4.0//EN"/
to 
<xsl:output

    And removed the doctype declartion in the above xslt

  2. My version of casperjs was setting xml namespace to

    xmlns="http://www.w3.org/1999/xhtml

    After unsuccessfully trying to match the namespace in the xslt, I gave up and removed the namespace from the xunit xml, by

    sed -i 's@ xmlns="http://www.w3.org/1999/xhtml" @ @' "output.xml"
  3. Inserted the style into xml so can be rendered in the browser

    sed -i 's@ encoding="UTF-8"?>@ encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="nosetests.xslt" ?>@' "output.xml"
  4. Removed timestamp info from the xslt as it was in UTC timezone and could be very confusing when looking at the results (attempts to convert to my TZ in client were unsuccessful)