Wednesday, January 20, 2016

Removing multiple items from a JavaScript array, no for loops

So I was reviewing some code today that was using two for loops to remove elements from a JavaScript array, I won't relate the code here but I figured there as got to be a nicer way. So the first guess is to make use of the filter method; but that creates a new array which might be problem if the array is large but not if it is fairly small or not called very often.

providers = providers.filter(function(provider) {
     return provider.age > 35
}

For performance or API reasons you might want to perform an in place removal using splice, so you can simple map the values to indexes you want to remove, reverse the order then perform a sequence of splice operations:

providers.map(function(provider, index) {
    return provider.age > 35 ? index : -1;
}).filter(function(index) {
    return index >= 0;
}).reverse().forEach(function(index) {
    providers.splice(index,1);
});

You can play with this code in this jsfiddle.

Update 21 Jan 20125: The hazard of doing full stack development is that it is easy to accidentally duck type between languages whilst forgetting the impact. In Java the equivalent streaming code using map just returns another lazy step in the stream whereas in JavaScript you end up creating a new array of the same size as the original. This removes the performance improvement over the first example if you have lots of data. This is simple too fix though as you can use the reduce function to just create an array and populate it using the reduce operation:

providers.reduce(function(list, provider, index) {
    if (provider.age> 35) list.push(index);
    return list;
}, []).reverse().forEach(function(index) {
    providers.splice(index,1);
});

Here is the updated fiddle. I guess at some point I am going to have to benchmark these variants to see which is best; but that is for a less busy day.

Tuesday, January 12, 2016

Beware of slightly different behaviour between ConcurrentMap and ConcurrentHashMap for computeIfAbsent

So I was refactoring some code to use non-locking collection classes and I noticed the this significant different between ConcurrentMap.computeIfAbsent and ConcurrentHashMap.computeIfAbsent. They key different is that for the former default implementing the mapping function can be called many times for a particular key where as for the concrete implementation it will be called only once. This will likely affect whether the code in the function needs to be thread safe or not.

Right okay so armed with that you know that any collection that implements just ConcurrentMap will inherit this behaviour, further to that I found in particular guava would return a different implementation depending on the passed on parameters:

ConcurrentMap map1 = new MapMaker().makeMap();
System.out.println(map1.getClass());

ConcurrentMap map2 = new MapMaker().weakKeys().makeMap();
System.out.println(map2.getClass());

....

class java.util.concurrent.ConcurrentHashMap
class com.google.common.collect.MapMakerInternalMap

MapMakerInternalMap doesn't override computerIfAbsent therefore the behaviour of this function will be significantly different depending on the parameters to pass into the maker, something that might be apparent from the get go.

Wednesday, November 25, 2015

A very simple implementation of Suite to make it easy to run across a bunch of test machines at the same time.

So I was presented with a large suite that I need to distribute amongst an arbitrary number of Hudson slaves. Now my first guess would be to hand balance this into N sub suites; but that would require on-going work and would need to change depending on the number of nodes. (We have a target of 30 minutes to run all tests in parallel which is going to take some machines)

@RunWith(Suite.class)
@Suite.SuiteClasses({
    TableCRUD.class,
    TableCRUDChild.class,
    TableFilterSort.class,
    TableDefaultQuery.class,
    TableActions.class,
    ....
})
public class DevSuite {

}

So the quickest way I found to do this was to write different version of Suite that allocated the tests to different bins. This implementation doesn't ensure properly balanced bins as it is biased towards keeping the order stable and consistent which is useful for comparing tests results split into multiple jobs.

import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;
import java.util.function.Function;
import java.util.function.ToIntFunction;
import java.util.stream.Collector;
import java.util.stream.Collectors;
import org.junit.runner.Runner;
import org.junit.runners.Suite;
import org.junit.runners.model.InitializationError;
import org.junit.runners.model.RunnerBuilder;

/**
 * Try to proportion a suites tests into "SPLIT" number of equals sections and
 * then just run "SECTION" rather than all of the tests. Defined by
 * "SplitSuite.split" and "SplitSuite.section" respectively
 */
public class SplitSuite
        extends Suite {

    private static int SECTIONS
            = Integer.getInteger("SplitSuite.split", -1);
    private static int SECTION
            = Integer.getInteger("SplitSuite.section", -1);

    public SplitSuite(RunnerBuilder builder, Class<?>[] classes) throws InitializationError {
        super(builder, classes);
    }

    public SplitSuite(Class<?> klass, RunnerBuilder builder) throws InitializationError {
        super(klass, builder);
    }

    public List<Runner> getChildren() {
        if (SECTIONS < 0) {
            return super.getChildren();
        } else if (SECTIONS == 0) {
            throw new IllegalArgumentException("SplitSuite.split needs to be a positive integer, it cannot be zero. A negative valid will disable this feature.");
        } else if (SECTION >= SECTIONS) {
            throw new IllegalArgumentException("SplitSuite.section parameter " + SECTION + " needs to be less than SplitSuite.split " + SECTIONS);
        }

        final Map<Integer, List<Runner>> collect = originalList.stream().collect(
                assignToGroups(SECTIONS, Runner::testCount, originalList));

        return collect.get(SECTION);
    }

    
    /**
     * Assigns object to buckets, moving to the next when filled, whilst preserving
     * the original order.
     * @param <Type> The type that has some kind of size property
     * @param sections The number of sections to split the code across
     * @param sizer A function to return the size of the given object
     * @param originalList The original list to provide a base line size
     * @return A map that contains an order list of sections
     */
    public static <Type> Collector<Type, ?, Map<Integer, List<Type>>> assignToGroups(
            int sections, 
            final ToIntFunction<Type> sizer,
            Collection<Type> originalList) {

        // Get length
        int count = originalList.stream().collect(Collectors.summingInt(sizer));
        // Workout section length
        int sectionLength = count / sections;
        
        Function<Type, Integer> grouping = new Function<Type, Integer>() {
            
            int counter = 0;
            
            @Override
            public Integer apply(Type t) {
                int section = Math.min(counter / sectionLength, sections - 1);
                counter += sizer.applyAsInt(t);                
                return section; 
            }
        };
        
        
        return Collectors.groupingBy(grouping, TreeMap::new, Collectors.toList());
    }
    
    
}



So a quick change to the suite class....

@RunWith(SplitSuite.class)
@Suite.SuiteClasses({
    TableCRUD.class,
    TableCRUDChild.class,
    TableFilterSort.class,
    TableDefaultQuery.class,
    TableActions.class,
    ....
})
public class DevSuite {

}

Now the actual test step looks like this, note that this is running under hudson but you should be able to achieve something similar on the CI server of your choice. In order to make my life easier I derive the section number from the job name for so 3 sections I have ..._0 ..._1 and ..._2. In hudson the later jobs all cascade from the the first sharing all properties which makes maintenance a lot easier.

java ... -DSplitSuite.sections=${SECTIONS} -DSplitSuite.section=`echo ${JOB_NAME} | sed -e s/.*_//` ...

And it appears to work, obviously the algorithm to split the suites could be better to produce more balanced results; but that is an effort for another day.....

Friday, July 31, 2015

Creating a shared library for Jersey 2.19 to use with Weblogic 12.1.3

Weblogic server comes with a shared library so you can deploy JAX-RS 2.0 applications; but is limited to Jersey version 2.5.1 and the instructions for changing this are not entirely obvious or straightforward. I have recently joined a new team at Oracle and one of the first things I did was to look at upgrading the dependent libraries. Now I have talked to the Jersey team and they don't support this combination; but it maybe useful enough to get you out of a bind until the next version of Weblogic is released.

I am going to do this using Maven because it means all the packaging and downloading is done for you. The basic structure of the project is as follows:

|-pom.xml
|-src
| |-main
| | |-java
| | |-resources
| | | |-META-INF
| | | | |-MANIFEST.MF
| | |-webapp
| | | |-WEB-INF
| | | | |-web.xml
| | | | |-weblogic.xml

I just generated a vanilla Maven project using Netbeans and then added in the Jersey dependencies I needed, it is likely this file could be cut down a little bit more with some determination. But it worked well enough for me:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    

    <groupId>com.example</groupId>
    <artifactId>JerseyLibrary</artifactId>
    <version>2.19</version>
    <packaging>war</packaging>

    <name>Jersey Library</name>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
         
        <jersey.version>2.19</jersey.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.glassfish.jersey.containers</groupId>
            <artifactId>jersey-container-servlet</artifactId>
            <version>${jersey.version}</version>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.core</groupId>
            <artifactId>jersey-client</artifactId>
            <version>${jersey.version}</version>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.media</groupId>
            <artifactId>jersey-media-multipart</artifactId>
            <version>${jersey.version}</version>
            <type>jar</type>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.media</groupId>
            <artifactId>jersey-media-moxy</artifactId>
            <version>${jersey.version}</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.1</version>
                <configuration>
                    <source>1.7</source>
                    <target>1.7</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-war-plugin</artifactId>
                <version>2.3</version>
                <configuration>
                    <failOnMissingWebXml>false</failOnMissingWebXml>
                    <archive>
                        <manifestFile>src/main/resources/META-INF/MANIFEST.MF</manifestFile>
                    </archive>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <version>2.6</version>
                <executions>
                    <execution>
                        <phase>validate</phase>
                        <goals>
                            <goal>copy</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>${endorsed.dir}</outputDirectory>
                            <silent>true</silent>
                            <artifactItems>
                                <artifactItem>
                                    <groupId>javax</groupId>
                                    <artifactId>javaee-endorsed-api</artifactId>
                                    <version>7.0</version>
                                    <type>jar</type>
                                </artifactItem>
                            </artifactItems>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>

The next most important file in the MANIFEST.MF file, this tells Weblogic when you try to deploy the .war that this is a library and also contains the versions it supplies:

Specification-Title: Weblogic JAX-RS 2.0
Specification-Version: 2.0
Implementation-Title: Weblogic JAX-RS 2.0 Reference Implementation
Implementation-Version: 2.19
Extension-Name: jax-rs

Finally, you have to include a weblogic.xml file to tell the server that some classes you need to take from here rather than the server class loader. I got the basis of this from the file that comes with the 2.5.1 shares library that ships with 12.1.3 and then added a few more lines to take in account how the code has moved on. Depending on what your code is doing you may have to add a few more.

<?xml version="1.0" encoding="UTF-8"?>
<weblogic-web-app
        xmlns="http://xmlns.oracle.com/weblogic/weblogic-web-app"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://xmlns.oracle.com/weblogic/weblogic-web-app http://xmlns.oracle.com/weblogic/weblogic-web-app/1.4/weblogic-web-app.xsd">

  <container-descriptor>
        <prefer-application-packages>
            <!-- apis -->
            <package-name>javax.ws.rs.*</package-name>

            <!-- guava -->
            <package-name>com.google.common.*</package-name>

            <!-- jersey providers -->
            <package-name>com.sun.jersey.*</package-name>
            <package-name>org.glassfish.jersey.*</package-name>
            
            <package-name>jersey.repackaged.*</package-name>
            
            <!-- hk2 -->
            <package-name>org.jvnet.hk2.*</package-name>
            <package-name>org.jvnet.hk2.tiger_types.*</package-name>
            <package-name>org.glassfish.hk2.*</package-name>

            <package-name>javassist.*</package-name>

            <!-- media providers -->
            <package-name>org.eclipse.persistence.jaxb.rs.*</package-name>
            <package-name>org.codehaus.jackson.jaxrs.*</package-name>

            <!-- wls -->
            <package-name>weblogic.jaxrs.api.client.*</package-name>
            <package-name>weblogic.jaxrs.internal.api.client.*</package-name>
            <package-name>weblogic.jaxrs.dispatch.*</package-name>
            <package-name>weblogic.jaxrs.monitoring.util.*</package-name>
        </prefer-application-packages>
    </container-descriptor>
</weblogic-web-app>

Now all this needs is a quick mvn install and in the target directory there will be a nice complete shared library called JerseyLibrary-2.19.war that you can deploy in the normal way. Remember of course to update the library entries for the .war that is going to depend on this to have the right versions in it so it does pick up the 2.5.1 version.

Monday, March 9, 2015

Quick and dirty test for thread leakage in java

I was looking at a bit of code was leaking class loaders when deployed to tomcat - so I wanted to quick rule out that a background Thread was holding onto the loader. Pain by eye, it is quite easy with a few unix commands to process the stack trace, filter for the lines that contain the thread name, filter for just the thread names and sort:
sudo -u tomcat jstack 1853 | grep " prio=" | sed -e 's/"\(.*\)".*/\1/' | sort
Once this is done it is a simple matter to pipe the output to a file, say /tmp/before.threads, then run the deploy task a small prime number of times in order to create a clear signal, and then compare with the original thread list
[gdavison ~]$ diff /tmp/before.threads /tmp/after.threads
26a27
> RMI TCP Connection(11)-127.0.0.1
[gdavison ~]$
As with any server there will be a lot going on, in this case I saw a spurious RMI thread we can probably ignore, but if there was any application specific thread leakage the problem should repeat a number of times that is cleanly divisible with the prime number you picked before, for example where the prime is three with an example issue:
[gdavison@slc01htu ~]$ diff /tmp/before.threads /tmp/after.threads
8a9,11
> Problem Thread
> Problem Thread
> Problem Thread
26a30
> RMI TCP Connection(11)-127.0.0.1
[gdavison ~]$

As I say quick and dirty; but might be worth writing down for later use.

Thursday, February 12, 2015

How long did I sleep last night: using Cordova, HealthKit, and JavaScript, and a handful of Promises

Now that we can know how much activity I did yesterday, we can look at whether I am getting enough sleep.

A quick look at the Apple documentation lets us see that HKCategoryTypeIdentifierSleepAnalysis is the right measure to be using. This is a subtype of SampleType so we can use the querySampleType function.

First of all here is the same boilerplate code, with the minor modification to allow for access to the sleep data rather than the step data we used before. You could of course combine the two.

var wearable = {

    avaliable: _.once(function() {

        return new Promise(function(resolve, reject) {
        
            if (window.plugins.healthkit) {
                window.plugins.healthkit.available(
                        function() {
                            console.log("Healthkit is avaliable");
                            resolve(true);
                        },
                        function() {
                            console.log("Healthkit is not avaliable");
                            reject(Error(false));
                        }
                );
            } else {
                reject(Error("HealthKit Not Available"));
            }
        });
    }),


    getHealthKit : _.once(function() {

        return wearable.avaliable().then(function() {

            return new Promise(function(resolve, reject) {
                window.plugins.healthkit.requestAuthorization(
                        {
                            'readTypes': ['HKCategoryTypeIdentifierSleepAnalysis'],
                            'writeTypes': []
                        },
                function() {
                    console.log("HealthKit authorisation accepted");
                    resolve(window.plugins.healthkit);
                },
                function() {
                    reject(Error("HealthKit authorisation rejected"));
                });
            });
        });
    }),

    ...

}



Then it is s simple matter of querying HeathKit with a useful date range. Note I use limit and ascending to access the last data point - it is possible that different tools will report multiple entires for a single night. (In particular new parents) I need to do a little bit more investigation here.

wearable = {

    ... 

    querySleep: function() {

        wearable.getHealthKit().then(function(healthkit) {

                var startDate = moment().subtract('d', 1).toDate();

                healthkit.querySampleType({
                        'startDate': startDate,
                        'endDate': moment().toDate(),
                        'sampleType': "HKCategoryTypeIdentifierSleepAnalysis",
                        'ascending': "NO",
                        'limit': 1
                    },
                    function(value) {
                        console.log("Success for runing sleep query");

                        // Debug output for the momment
                        value.forEach(function(next) {

                            console.dir(next);
                            var measure = next.value === 1 ? "Sleeping" : "In Bed";
                            var minutesSleep = moment(next.endDate).diff(next.startDate, "minutes");
                            var hoursSleep = moment(next.endDate).diff(next.startDate, "hours");
                            console.log("Entry got " + minutesSleep + " minutes " + measure);
                            console.log("Entry got " + hoursSleep + " hours " + measure);

                        });

                        // Use data in some way
                    });

            }
        }
    },
}


Finally it would be really useful to be able to be told when data is added to HealthKit by another app for example the client app for a wearable. You can do this using the monitorSampleType function. Ideally this would code would then use something called an anchored query so you only access the most recently added data; but I haven't had time to implement this yet so we simply call querySleep().

Now making sure you app wakes up in the background to receive this data is a whole other blog....

var wearable = {

    ...

    monitorSleep: _.once(function() {

        console.log("Starting to monitor sleep");


        wearable.getHealthKit().then(function(healthkit) {
            healthkit.monitorSampleType({
                    'sampleType': "HKCategoryTypeIdentifierSleepAnalysis"
                },
                _.debounce(function(value) {
                    console.log("Sleep data has been updated, lets see if anything interesting has been added");
                    wearable.querySleep();
                }, 2000),
                function() {
                    console.log("Failed to monitor sample data");
                    console.dir(arguments);
                });

        });
    })
}




Tuesday, February 10, 2015

Per client cookie handling with Jersey

A lot of REST services will use cookies as part of the authentication / authorisation scheme. This is a problem because by default the old Jersey client will use the singleton CookieHandler.getDefault which is most cases will be null and if not null will not likely work in a multithreaded server environment. (This is because in the background the default Jersey client will use URL.openConnection)

Now you can work around this by using the Apache HTTP Client adapter for Jersey; but this is not always available. So if you want to use the Jersey client with cookies in a server environment you need to do a little bit of reflection to ensure you use your own private cookie jar.

final CookieHandler ch = new CookieManager();
    
    Client client = new Client(new URLConnectionClientHandler(
       new HttpURLConnectionFactory() {

        @Override
        public HttpURLConnection getHttpURLConnection(URL uRL) throws IOException {
            HttpURLConnection connect = (HttpURLConnection) uRL.openConnection();
        
            try {
                
                Object toModify = connect;
                
                if (!(toModify instanceof sun.net.www.protocol.http.HttpURLConnection)) {
        
                    Field delegateField = connect.getClass().getDeclaredField("delegate");
                    delegateField.setAccessible(true);
                    toModify = MethodHandles.lookup().unreflectGetter(delegateField)
                        .bindTo(toModify).invoke();
                    
                }
        
                Field cookieField = sun.net.www.protocol.http.HttpURLConnection.class.getDeclaredField("cookieHandler");
                cookieField.setAccessible(true);
                MethodHandle mh = MethodHandles.lookup().unreflectSetter(cookieField);
                mh.bindTo(toModify).invoke(ch);
            } catch (Throwable e) {
                e.printStackTrace();
            }
        
            return connect;

        }
    }));
    

This will only work if your environment is using the internal implementation of sun.net.www.protocol.http.HttpURLConnection that comes with the JDK. This appears to be the case for modern versions of WLS.

For JAX-RS 2.0 you can do a similar change using Jersey 2.x specific ClientConfig class and HttpUrlConnectorProvider.

final CookieHandler ch = new CookieManager();


    Client client =
        ClientBuilder.newClient(new ClientConfig().connectorProvider(new HttpUrlConnectorProvider().connectionFactory(new HttpUrlConnectorProvider.ConnectionFactory() {
            @Override
            public HttpURLConnection getConnection(URL uRL) throws IOException {
                HttpURLConnection connect = (HttpURLConnection) uRL.openConnection();
            
                try {
                    
                    Object toModify = connect;
                    
                    if (!(toModify instanceof sun.net.www.protocol.http.HttpURLConnection)) {
            
                        Field delegateField = connect.getClass().getDeclaredField("delegate");
                        delegateField.setAccessible(true);
                        toModify = MethodHandles.lookup().unreflectGetter(delegateField)
                            .bindTo(toModify).invoke();
                        
                    }
            
                    Field cookieField = sun.net.www.protocol.http.HttpURLConnection.class.getDeclaredField("cookieHandler");
                    cookieField.setAccessible(true);
                    MethodHandle mh = MethodHandles.lookup().unreflectSetter(cookieField);
                    mh.bindTo(toModify).invoke(ch);
                } catch (Throwable e) {
                    e.printStackTrace();
                }
            
                return connect;

            }
        })));



Update 11th Feb 2015: It seems in some cases, in particular using https, I have seen the HttpURLConnection wrapped in another class, to work around this just use reflection to access the value of the delegate field. I have updated the code examples to reflect this issue.