1. UnicastProcessor

    UnicastProcessor hotSource = UnicastProcessor.create()
    hotSource.subscribe(System.out::println)
    hotSource.onNext("foo");
    hotSource.onNext("bar");
    . . .
    0

    Add a comment

  2. Problem:
    You have a large string (string A) and another string (string B) and you want to find out if all characters of string B are present in string A. The problem is not complex through a basic loop and ‘contains’ call, but could be made more optimal by splitting both strings. Why splitting is important? By splitting string A you creating a potential to discover that all character in string B are present in split from String A and you can break off the loop and return true. If not you concatenate first split with the second one and repeat. The bottom line is that you only have to look through the whole string if you can’t find a match until the last split. Splitting string B is also important since if a split from string B was matched to a string A, you no longer have to include it in your matching logic.

    public class CharSearchInString {

        public static void main(String[] args) {
            String[] driverStrings = "java.util.concurrent.atomic.AtomicInteger".split("(?<=\\G.{4})");
            String[] comparableStrings = "avrutcimIr".split("(?<=\\G.{3})");
            // should return true as all characters in 'comparableStrings' are
            // present in 'driverStrings'
            System.out.println(performCompare(driverStrings, comparableStrings));
        }

        /**
         *
         */
        private static boolean performCompare(String[] driverStrings, String[] comparableStrings) {
            int csCount = 0;
            String driverString = "";
            for (int dsCount = 0; dsCount < driverStrings.length;) {
                innerLoop:
                for (; csCount < comparableStrings.length;) {
                    String comparableString = comparableStrings[csCount];
                    if (compares(driverString, comparableString)) {
                        csCount++;
                    } else {
                        driverString += driverStrings[dsCount];
                        dsCount++;
                    }
                    break innerLoop;
                }
                if (csCount == comparableStrings.length) {
                    break;
                }
            }
            // will return true if all strings in 'comparableStrings' were matched
            // to at least one string in 'driverStrings'
            return csCount == comparableStrings.length;
        }

         /**
          *
          */
        private static boolean compares(String a, String b) {
            byte[] bArray = b.getBytes();
            for (int i = 0; i < bArray.length; i++) {
                byte x = bArray[i];
                if (!a.contains((char) x + "")) {
                    return false;
                }
            }
            return true;
        }

    }

    0

    Add a comment



  3. public static void setFinalField(Field field, Object newValue) throws Exception {
       field.setAccessible(true);
       Field modifiersField = Field.class.getDeclaredField("modifiers");
       modifiersField.setAccessible(true);
       modifiersField.setInt(field, field.getModifiers() & ~Modifier.FINAL);
       field.set(null, newValue);
    }

    0

    Add a comment



  4. Thread.currentThread().getStackTrace()

    0

    Add a comment


  5. Java 8 introduces type intersection allowing object to be cast to an intersection of types
    public class TypeIntersectionSample {
        public static void main(String... arg) {
            Serializable bser = toSerializable(Integer::sum);
        }
        public static <T extends BinaryOperator<Integer> & Serializable> T toSerializable(T bo) {
            return (T) bo;
        }
    }


    0

    Add a comment



  6. There is a new factory method in ThreadLocal allowing you to create and set initial value all at he same time via lambda expression.


    ThreadLocal tl = ThreadLocal.withInitial(() -> "Hello");

    The rest is the same as before with regard to set(..)/get().

    1

    View comments


  7. Simply use MemoryMXBean

    MemoryMXBean memoryBean = ManagementFactory.getMemoryMXBean();
    MemoryUsage heapUsage = memoryBean.getHeapMemoryUsage();
    MemoryUsage nonHeapUsage = memoryBean.getNonHeapMemoryUsage();

    Then simply interrogate instance of MemoryUsage

    long usedMemory = heapUsage.getUsed();
    long maxMemory = heapUsage.getMax();
    long initMemory = heapUsage.getInit();
    long maxMemory = heapUsage.getMax();

    The same goes for non heap 


    0

    Add a comment


  8. What is it?
    Mesos is an independent cluster platform which hosts cluster computing frameworks.
    Giving the fact that cluster platform's main responsibility to manage cluster resources a good example of similar architecture in Hadoop would be YARN and Tez, where YARN is native Hadoop cluster platform with Tez being domain-specific (domain being DAG) cluster computing framework hosted by YARN. This allows not only multi-framework hosting but also multi-instances and/or versions of the same framework hosted in the same shared cluster.

    Architecture
    Mesos - Consists of Master and Slave nodes with allocation modules hosted by Master
    Framework - Consists of Scheduler and Executor processes launched on every node (to execute framework's tasks)

    Scheduling and data locality
    Mesos introduces the concept of decentralized scheduler. The arguments for that are:

    • the need for expressive API to support all the frameworks (known and un-known)
    • new frameworks may already come with its own scheduling mechanism, so refactoring it into centralized would not be feasible

    So Mesos delegates scheduling to the frameworks themselves via resource offers. Mesos offers resources to frameworks based on organizational policies (e.g., fair, priority etc), while the framework itself decides to accept/reject or how much of resources to accept.

    Slave reports resource availability to the Master -> 
    Master(4 cpu and 8Gb on this slave) via allocation policy module -> 
    Framework's scheduler (takes 2 CPU, 2GB mem, on this slave or rehect) -> 
    Master(OK, and launches the tasks on such slave)

    Unused resources are offered to other frameworks and the overall process repeats upon completion of tasks.

    This is one of the mechanism frameworks utilize to achieve data locality - by accepting and/or rejecting resource offers from slaves based on data proximity.

    So the main design philosophy for Mesos - define a minimal interface that enables efficient resource sharing across frameworks, and otherwise push control of task scheduling and execution to the frameworks.

    Allocation module
    Based on assumption that task have a short life span which means it can reallocate resources as quick as tasks complete. However in a cases of long-running tasks Mesos can revoke (kill) tasks by killing task itself and/or killing executor if task does not respond. It's all in implementation detail of allocation module policies.
    While MR based frameworks can easily live with re-allocated/killed tasks, MPI-style frameworks can't, so allocation modules can expose guaranteed allocation to each framework - basically a set of resources permanently taken by the framework.

    Since resource acceptance is based on accept or reject mechanism, rejection in itself may introduce an unnecessary chatter especially for cases where certain slaves would always reject a particular scheduling request. Mesos allows such chatter to be further reduced by providing allocation filters to the master essentially allowing resource offer not to go out to a particular slave if it does not pass a filter criteria

    Resource offers will be rescinded if a framework does not respond timely to resource offer, thus allowing such resources to be re-offered to other frameworks.

    Performance Isolation
    Performance isolation achieved via native mechanisms (e.g., Linux containers, etc.)

    Master fault tolerance (hot stand-by)
    Master has few standby nodes. In a case of master failure ZooKeeper will select a new master from the list. Since all (hot and standby) masters receive events from the cluster, a new master will be able to reconstruct its latest state from these events.


    Opinion:
    Mesos may have its benefits in heterogeneous clusters often seen in small/medium sized businesses where maintaining a domain specific cluster is not cost efficient due to low load and high idle scenarios, so a single cluster platform can host multiple cluster computing frameworks sharing resources between them, thus limiting the idle time.
    However, in large scale high load data clusters (e.g., Hadoop) having a native cluster platform such as YARN is more beneficial based on a simple premise that specialized solutions are generally better then generalized.
    Having said that, YARN is perfectly capable of hosting cluster computation frameworks which may have nothing to do with Hadoop all together, thus allowing Hadoop/YARN cluster to be used as heterogeneous cluster compute platform.

    Missrepresentation:
    In Mesos Architecture white paper (http://mesos.berkeley.edu/mesos_tech_report.pdf). In Page 2 Section 2 it talks about an example of what needs to happen to data sitting in Hadoop after MR processing if such data needs to be integrated with MPI. It talks about requirement for spinning up a new MPI cluster and moving terrabytes of data from Hadoop cluster while making the argument that MPI could be just another compute framework hosted in the same cluster.
    That is actually not true and these capabilities existed in Hadoop even before YARN/Tez, since MPI concepts could always be implemented as part of your MR job (although I'd admit it was not simple). However with YARN and Tez it is relatively simple. In fact Tez is being more flexible when it comes to this sort of requirements since it is a low-level DAG framework comprised of simple Vertexes "processors" (filters) connected by Edges (pipes) as in standard pipes-and-filters architecture.


    0

    Add a comment


  9. When learning new constructs of Java 8 you quickly learn about method references (or functional references). The similarity is also often drawn to object references. This may lead to one of these mistakes:

    Object function = File::isHidden;

    The above will result in compile error "The target type of this expression must be a functional interface".
    The correct way is to use the appropriate functional interface:

    Predicate<File> function = File::isHidden;

    0

    Add a comment


  10. Variety of exceptions could be categorized as class loading exceptions, since their root cause most often leads one to discover issues with the classpath. Some of these are NoClassDefFoundError, LinkageError, NoSuchMethodError etc. Debugging these issues is greatly simplified when you know where a class is loaded from (i.e., which JAR or directory).
    The code below will extract the URL for the location of the source for a particular class (Foo in this case)

    URL fooClassSourceUrl = Foo.class.getProtectionDomain().getCodeSource().getLocation()


    0

    Add a comment

Loading