concurrent

https://github.com/kiwanami/emacs-deferred.git

git clone 'git://github.com/kiwanami/emacs-deferred.git'
138

deferred.el # travis badge

‘deferred.el’ provides facilities to manage asynchronous tasks. The API and implementations were translated from JSDeferred (by cho45) and Mochikit.Async (by Bob Ippolito) in JavaScript.

Installation

You can install deferred.el from MELPA by package.el.

Sample codes

You can find following sample codes in ‘deferred-sample.el’. Executing ‘eval-last-sexp’ (C-x C-e), you can try those codes.

Basic usage

This is a basic deferred chain. This code puts some outputs into message buffer, and then require a number from minibuffer.

Chain:

(deferred:$
  (deferred:next
    (lambda () (message "deferred start")))
  (deferred:nextc it
    (lambda ()
      (message "chain 1")
      1))
  (deferred:nextc it
    (lambda (x)
      (message "chain 2 : %s" x)))
  (deferred:nextc it
    (lambda ()
      (read-minibuffer "Input a number: ")))
  (deferred:nextc it
    (lambda (x)
      (message "Got the number : %i" x)))
  (deferred:error it
    (lambda (err)
      (message "Wrong input : %s" err))))

Timer

After evaluating this s-exp and waiting for 1 second, a message is shown in the minibuffer.

Timer:

(deferred:$
  (deferred:wait 1000) ; 1000msec
  (deferred:nextc it
    (lambda (x)
      (message "Timer sample! : %s msec" x))))

Commands and Sub-process

This s-exp inserts the result that is performed by the command ‘ls -la’. (This s-exp may not run in windows. Try ‘dir’ command.)

Command process:

(deferred:$
  (deferred:process "ls" "-la")
  (deferred:nextc it
    (lambda (x) (insert x))))

HTTP GET : Text

This s-exp inserts a text from http://www.gnu.org asynchronously. (You can clear the result with undo command.)

HTTP GET:

(require 'url)

(deferred:$
  (deferred:url-retrieve "http://www.gnu.org")
  (deferred:nextc it
    (lambda (buf)
      (insert  (with-current-buffer buf (buffer-string)))
      (kill-buffer buf))))

HTTP Get : Image

This s-exp inserts an image from google asynchronously.

Get an image:

(deferred:$
  (deferred:url-retrieve "http://www.google.co.jp/intl/en_com/images/srpr/logo1w.png")
  (deferred:nextc it
    (lambda (buf)
      (insert-image
       (create-image
        (let ((data (with-current-buffer buf (buffer-string))))
          (substring data (+ (string-match "\n\n" data) 2)))
        'png t))
      (kill-buffer buf))))

Parallel

This s-exp retrieves two images from google concurrently and wait for the both results. Then, the file sizes of the images are inserted the current buffer.

Parallel deferred:

(deferred:$
  (deferred:parallel
    (lambda ()
      (deferred:url-retrieve "http://www.google.co.jp/intl/en_com/images/srpr/logo1w.png"))
    (lambda ()
      (deferred:url-retrieve "http://www.google.co.jp/images/srpr/nav_logo14.png")))
  (deferred:nextc it
    (lambda (buffers)
      (loop for i in buffers
            do
            (insert
             (format
              "size: %s\n"
              (with-current-buffer i (length (buffer-string)))))
            (kill-buffer i)))))

Deferred Combination : try-catch-finally

This s-exp executes following tasks: * Getting an image by wget command, * Resizing the image by convert command in ImageMagick, * Insert the re-sized image into the current buffer. You can construct the control structure of deferred tasks, like try-catch-finally in Java.

Get an image by wget and resize by ImageMagick:

(deferred:$

  ;; try
  (deferred:$
    (deferred:process "wget" "-O" "a.jpg" "http://www.gnu.org/software/emacs/tour/images/splash.png")
    (deferred:nextc it
      (lambda () (deferred:process "convert" "a.jpg" "-resize" "100x100" "jpg:b.jpg")))
    (deferred:nextc it
      (lambda ()
        (clear-image-cache)
        (insert-image (create-image (expand-file-name "b.jpg") 'jpeg nil)))))

  ;; catch
  (deferred:error it ;
    (lambda (err)
      (insert "Can not get a image! : " err)))

  ;; finally
  (deferred:nextc it
    (lambda ()
      (deferred:parallel
        (lambda () (delete-file "a.jpg"))
        (lambda () (delete-file "b.jpg")))))
  (deferred:nextc it
    (lambda (x) (message ">> %s" x))))

Here is an another sample code for try-catch-finally blocks. This is simpler than above code because of the ‘deferred:try’ macro. (Note: They bring the same results practically, but are not perfectly identical. The ‘finally’ task may not be called because of asynchrony.)

Try-catch-finally:

(deferred:$
  (deferred:try
    (deferred:$
      (deferred:process "wget" "-O" "a.jpg" "http://www.gnu.org/software/emacs/tour/images/splash.png")
      (deferred:nextc it
        (lambda () (deferred:process "convert" "a.jpg" "-resize" "100x100" "jpg:b.jpg")))
      (deferred:nextc it
        (lambda ()
          (clear-image-cache)
          (insert-image (create-image (expand-file-name "b.jpg") 'jpeg nil)))))
    :catch
    (lambda (err) (insert "Can not get a image! : " err))
    :finally
    (lambda ()
      (delete-file "a.jpg")
      (delete-file "b.jpg")))
  (deferred:nextc it
    (lambda (x) (message ">> %s" x))))

Timeout

Although a long time command is executed (3 second sleeping), the task is rejected by timeout for 1 second.

The function deferred:earlier also runs asynchronous tasks concurrently, however, the next deferred task receives the first result. The other results and tasks will be rejected (canceled or ignored).

Timeout Process:

(deferred:$
  (deferred:earlier
    (deferred:process "sh" "-c" "sleep 3 | echo 'hello!'")
    (deferred:$
      (deferred:wait 1000) ; timeout msec
      (deferred:nextc it (lambda () "canceled!"))))
  (deferred:nextc it
    (lambda (x) (insert x))))

Here is an another sample code for timeout, employing ‘deferred:timeout’ macro.

Timeout macro:

(deferred:$
  (deferred:timeout
    1000 "canceled!"
    (deferred:process "sh" "-c" "sleep 3 | echo 'hello!'"))
  (deferred:nextc it
    (lambda (x) (insert x))))

Note that the deferred:timeout and deferred:earlier just rejects the task result and does not stop the running task chains. Please see the document for deferred:cancel.

Loop and Animation

This s-exp plays an animation at the cursor position for few seconds. Then, you can move cursor freely, because the animation does not block Emacs.

Returning a deferred object in the deferred tasks, the returned task is executed before the next deferred one that is statically connected on the source code. (In this case, the interrupt task is dynamically connected.)

Employing a recursive structure of deferred tasks, you can construct a deferred loop. It may seem the multi-thread in Emacs Lisp.

Loop and animation:

(lexical-let ((count 0) (anm "-/|\\-")
              (end 50) (pos (point))
              (wait-time 50))
  (deferred:$
    (deferred:next
      (lambda (x) (message "Animation started.")))

    (deferred:nextc it
      (deferred:lambda (x)
        (save-excursion
          (when (< 0 count)
            (goto-char pos) (delete-char 1))
          (insert (char-to-string
                   (aref anm (% count (length anm))))))
        (if (> end (incf count)) ; return nil to stop this loop
            (deferred:nextc (deferred:wait wait-time) self)))) ; return the deferred

    (deferred:nextc it
      (lambda (x)
        (save-excursion
          (goto-char pos) (delete-char 1))
        (message "Animation finished.")))))

Wrapping asynchronous function

Let's say you have an asynchronous function which takes a callback. For example, dbus.el, xml-rpc.el and websocket.el has such kind of asynchronous APIs. To use such libraries with deferred.el, you can make an unregistered deferred object using deferred:new and then start the deferred callback queue using deferred:callback-post in the callback given to the asynchronous function. If the asynchronous function supports “errorback”, you can use deferred:errorback-post to pass the error information to the following callback queue.

In the following example, run-at-time is used as an example for the asynchronous function. Deferred.el already has deferred:wait for this purpose so that you don't need the following code if you want to use run-at-time.

(deferred:$
  (deferred:next
    (lambda ()
      (message "1")
      1))
  (deferred:nextc it
    (lambda (x)
      (lexical-let ((d (deferred:new #'identity)))
        (run-at-time 0 nil (lambda (x)
                             ;; Start the following callback queue now.
                             (deferred:callback-post d x))
                     x)
        ;; Return the unregistered (not yet started) callback
        ;; queue, so that the following queue will wait until it
        ;; is started.
        d)))
  ;; You can connect deferred callback queues
  (deferred:nextc it
    (lambda (x)
      (message "%s" (1+ x)))))

API

Functions

Basic functions

Utility functions

Wrapper functions

Primitive functions

Utility Macros

Execution and Connection

Firing

Some deferred functions can fire a deferred chain implicitly. Following functions register a deferred object with the execution queue to run asynchronously.

The deferred tasks those are created by ‘deferred:new’ are never called. Using this object, a deferred chain can pause to wait for other events. (See the source for ‘deferred:wait’.)

One can fire the chain before connecting. That is, deferred objects wait for connecting the subsequent task holding the result value. The functions ‘deferred:succeed’ and ‘deferred:fail’ create those waiting objects.

Static connection

The ‘static connection (statically connected)’ is a connection between deferred tasks on the source code. This is a basic usage for the deferred chain.

The static connection is almost equivalent to ordinary callback notation as an argument in the function declarations. The deferred notation is easy to read and write better than the callback one, because the sequence of asynchronous tasks can be written by the deferred notation straightforward.

Dynamic Connection

Returning a deferred object in the deferred tasks, the returned task is executed before the next deferred one that is statically connected on the source code. This is the ‘dynamic connection (dynamically connected)’.

Employing a recursive structure of deferred tasks, you can construct higher level control structures, such as loop.

Discussion

Some discussions of writing deferred codes.

Using lexical scope

Using the lexical scope macro, such as ‘lexical-let’, the deferred tasks defined by lambdas can access local variables.

lexical-let Ex.:

(lexical-let ((a (point)))
  (deferred:$
    (deferred:wait 1000)
    (deferred:nextc it
      (lambda (x)
        (goto-char a)
        (insert "here!")))))

If you write a code of deferred tasks without lexical scope macros, you should be careful with the scopes of each variables.

Excursion (Current status)

The ‘excursion’ functions those hold the current status with the s-exp form, such as ‘save-execursion’ or ‘with-current-buffer’, are not valid in the deferred tasks, because of execution asynchronously.

Wrong Ex.:

(with-current-buffer (get-buffer "*Message*")
  (deferred:$
    (deferred:wait 1000)
    (deferred:nextc it
      (lambda (x)
        (insert "Time: %s " x) ; 'insert' may not be in the *Message* buffer!
      ))))

In this case, using lexical scope macros to access the buffer variable, you can change the buffer in the deferred task.

Corrected:

(lexical-let ((buf (get-buffer "*Message*")))
  (deferred:$
    (deferred:wait 1000)
    (deferred:nextc it
      (lambda (x)
        (with-current-buffer buf ; Set buffer in the asynchronous task.
          (insert "Time: %s " x))))))

Be aware of return values

However the dynamic connection is a powerful feature, sometimes it causes bugs of the wrong execution order, because of returning not intended deferred objects.

Then, you should watch the return values of the deferred tasks not to cause an unexpected dynamic connection.

Debugging

The debugging of asynchronous tasks is difficult. Of course, you can use debugger for deferred tasks, but asynchronous tasks cause some troubles, such as interruptions of your debugging and timing gap of simultaneous deferred tasks. Therefore, logging is a safe debugging to observe the tasks correctly, for example, using the ‘message’ function and making custom application log buffer.

If deferred tasks fall into an infinite loop unexpectedly (but Emacs may not freeze), calling the command ‘deferred:clear-queue’, you can stop the deferred tasks immediately.

If the errors occurred in deferred tasks are caught by no errorback functions, finally the deferred framework catches it and reports to the message buffer. Because the implementation of the framework uses a ‘condition-case’ form, the debugger can not catch the signals normally. If you want to debug the errors in the deferred tasks with the debug-on-error mechanism, set the variable ‘deferred:debug-on-signal’ non-nil.

Wrapping a deferred task in the function ‘deferred:sync!’, you can wait for the result of the task synchronously. However, the wrapper function should be used for test or debug purpose, because the synchronous waiting is not exact.

Using macros

Writing deferred tasks with ‘deferred.el’, you may write a lot of ‘deferred:nextc’ and ‘lambda’ to define tasks. Defining a macro, you may write codes shortly. The test code ‘test-deferred.el’ uses many macros to shorten test codes.

On the other hand, using macros to hide ‘lambda’, it is difficult to realize when the deferred codes are evaluated. That is why ‘deferred.el’ does not provide lot of convenient macros. If you use macros, be careful evaluation timing of deferred forms.

Introduction for deferred

Following documents are good introduction to deferred.


(C) 2010, 2011 SAKURAI Masashi All rights reserved. m.sakurai at kiwanami.net