Ffidl

Extend Tcl without extensions

Ffidl

Ffidl allows you to call C functions using pure Tcl wrappers. You specify a function name, a library, a list of argument types, and a return type, and Ffidl takes care of the nasty details of converting a Tcl command call into a C function call for you. So, if you have a shared library and a specification of the entries in the library, you can wrap the library into a Tcl extension with Ffidl and pure Tcl.

The following example shows how to invoke libc's puts(3) function from Tcl code:

% package require Ffidl
0.8b0
% ffidl::callout cputs {pointer-utf8} int [ffidl::symbol libc.so.6 puts]
% cputs "Hello libc!"
Hello libc!
12

Ffidl can use the latest libffi (version 3.2.1 at the time of this writing) to dynamically construct calls to C functions from Tcl, to dynamically construct calls from C functions into Tcl. Libffi runs on many platforms, see Supported Platforms section on libffi's website for details.

Ffidl can also be configured (--with-libffcall) to use libffcall a GPL'ed foreign function package which implements both callouts and callbacks on many platforms. Versions 1.13 and 2.1 have been tested to work.

Ffidl uses Tcl's Tcl_LoadFile() and Tcl_FindSymbol(), to load dynamic libraries and discover the locations of functions.

Ffidl should be able to run on any system with a stubs enabled Tcl, libffi or libffcall support, and a libdl implementation.

Ffidl 0.8b0 is an alpha release. There are configuration details which you will need to attend to by hand in the current release. The initial development turned up a few bugs in libffi under linux-x86, so users on other architectures should be alert for similar problems. There are several open design issues still to be resolved, so there may be changes in the interfaces in future releases.

Assistance is needed to verify that the implementation works on all the architectures supported by libffi and libffcall, and the open design issues could use some discussion.

Changelog

Ffidl 0.8b0 (pre-release)

Ffidl 0.8b0 has the following changes:

The changes in 0.8b0 were implemented by Adrián Medraño Calvo for Patzschke + Rasp Software GmbH, and are under BSD License; do not contact the original Ffidl author for support about them.

Ffidl 0.7

Ffidl 0.7 has the following changes since 0.6:

The changes in 0.7 were implemented by Adrián Medraño Calvo for Patzschke + Rasp Software GmbH, and are under BSD License; do not contact the original Ffidl author for support about them.

Ffidl 0.6

Ffidl 0.6 has the following changes since 0.5:

The changes in 0.6 were implemented by Daniel A. Steffen, are Copyright © 2005 by Daniel A. Steffen and are under BSD License; do not contact the original Ffidl author for support about them.

Ffidl 0.6 has been verified to build and pass its testsuite on the following platforms:

Ffidl 0.5

The original Ffidl, created by Roger E Critchlow Jr. Please, refer to Ffidl 0.5 for details.

Commands, Functions, and Procs

Ffidl defines seven Tcl commands in the Ffidl package: ::ffidl::callout, ::ffidl::callback, ::ffidl::library, ::ffidl::symbol, ::ffidl::stubsymbol, ::ffidl::typedef, and ::ffidl::info; exports one function from the Ffidl shared library: ffidl_pointer_pun; and defines two helper procs in the Ffidlrt package in demos/ffidlrt.tcl: ::ffidl::find-lib, and ::ffidl::find-type, which are currently just stubs of their true form.

These interfaces should be considered subject to revision.

::ffidl::callout name {?arg_type1 ...?} return_type address ?protocol?
::ffidl::callout defines a Tcl command with the specified name which, when invoked, converts its arguments according to the arg_types specified, calls the function at the specified address, and converts the specified return_type into a Tcl result. The allowed types are described below.

The protocol specifies a calling convention to be used. The only platform which supports protocols is Windows where protocol may be cdecl, which is the default, or stdcall.

::ffidl::callback name {?arg_type1 ...?} return_type ?protocol? ?cmdprefix?
::ffidl::callback declares that a Tcl proc with the specified name will be invoked as a callback from C code. When invoked the arguments will be converted to Tcl values according to the arg_types specified, passed to name, the return value from name will be converted back into the specified return_type, and the value will be returned to the caller. The allowed types are described below. Returns a function pointer to the created callback.

The protocol specifies a calling convention to be used. The only platform which supports protocols is Windows where protocol may be cdecl, which is the default, or stdcall.

The cmdprefix specifies a command prefix to be invoked instead of the proc with the specified name. The arguments are appended to the command prefix before evaluation.

::ffidl::library ?-binding now|lazy? ?-visibility local|global? ?--? library
::ffidl::library load a dynamically linked library of name library. The option -binding determines whether the symbols bound on use or immediately. The option -visibility determines whether the bound symbols are available to further loaded libraries. Note that this options are not supported under all Ffidl configurations. When they are not specified, the platform's default for the Ffidl configuration is used.
::ffidl::symbol library symbol
::ffidl::symbol loads, if necessary, a dynamically linked library of name library and fetches the loaded address of symbol from the library. The kinds of symbols available vary with the implementation of dynamic loading.
::ffidl::stubsymbol library stubstable symbolnumber
::ffidl::stubsymbol returns the address of the symbol indexed by symbolnumber in the library's stubs table stubstable.

library can be one of tcl or tk and stubstable can be one of stubs, intStubs, platStubs, intPlatStubs or intXLibStubs.

::ffidl::typedef name type1 ?...?
::ffidl::typedef defines a new Ffidl type name. This may be either a simple alias for an existing type, or a list of types which form a structured aggregate. To pass a structure by value or return a structure by value, you must make a ::ffidl::typedef for it. But even if you only pass or receive structures by reference, you might want to define a structure in order to use the format, sizeof, and alignof options of ::ffidl::info on it.
::ffidl::info option ?...?
::ffidl::info implements a variety of information functions.
::ffidl::info alignof type
returns the alignment modulus for type.
::ffidl::info callbacks
returns a list of ::ffidl::callback declared names.
::ffidl::info callouts
returns a list of ::ffidl::callout defined names.
::ffidl::info canonical-host
returns the canonical host name as determined by autoconf.
::ffidl::info format type
returns a format string for type, in the style of the Tcl binary format and binary scan commands, using the correct endian format for integers and, for structures, including any pad bytes required for alignment of fields.
::ffidl::info have-int64
returns true if the host implements a 64 bit integer.
::ffidl::info have-long-double
returns true if the host implements the "long double" type.
::ffidl::info have-long-long
returns true if the host implements the "long long" type.
::ffidl::info interp
returns the current Tcl_Interp as an integer value.
::ffidl::info libraries
returns the list of libraries opened by ::ffidl::symbol.
::ffidl::info signatures
returns the list of function call signatures used by ::ffidl::callout.
::ffidl::info sizeof type
returns the size of type.
::ffidl::info typedefs
returns a list of ::ffidl::typedef defined names.
::ffidl::info use-callbacks
returns true if Ffidl was configured to use callbacks.
::ffidl::info use-libffcall
returns true if Ffidl was configured to use libffcall.
::ffidl::info use-libffi
returns true if Ffidl was configured to use libffi.
::ffidl::info use-libffi-raw
returns true if libffi implements the raw api.
::ffidl::info NULL
returns the null pointer value.
EXTERN void *ffidl_pointer_pun(void *pointer);
ffidl_pointer_pun is exported from the Ffidl shared lib to allow conversions between pointer representations to be coded as Ffidl bindings. There are some examples in ffidlrt.tcl.
::ffidl::find-lib library
::ffidl::find-lib converts a conventional name for a library into the path name for the library name appropriate to the host system. It is currently implemented in ffidlrt.tcl as a table lookup which returns the libraries appropriate to my Linux system.
::ffidl::find-type type
::ffidl::find-type converts a standard types such as size_t and time_t into real types appropriate to the host system. It is currently implemented in ffidlrt.tcl as a table lookup which returns the types appropriate to my Linux system.

Types

The Ffidl builtin types include the scalar C types in both their unsized forms and as explicitly bit sized types, and a variety of pointer treatments. Note that some types are only valid in certain contexts: arguments (arg), return (ret), or struct elements (elt).

In addition to the builtin types, the ::ffidl::typedef command may be used to define new types. Aliases for existing types may be used where ever the existing type may be used. Structured aggregates may be used as arguments, returns, or elements of other structures.

proc callback elt type definition
arg ret arg ret
- + - + - void void
+ + + + + char char
+ + + + + signed char signed char
+ + + + + unsigned char unsigned char
+ + + + + int int
+ + + + + unsigned unsigned int
+ + + + + short signed short int
+ + + + + unsigned short unsigned short int
+ + + + + long signed long int
+ + + + + unsigned long unsigned long int
+ + + + + float float
+ + + + + double double
+ + + + + long long long long
+ + + + + unsigned long long unsigned long long
+ + + + + long double long double
+ + + + + sint8 signed 8 bit int
+ + + + + uint8 unsigned 8 bit int
+ + + + + sint16 signed 16 bit int
+ + + + + uint16 unsigned 16 bit int
+ + + + + sint32 signed 32 bit int
+ + + + + uint32 unsigned 32 bit int
+ + + + + sint64 signed 64 bit int
+ + + + + uint64 unsigned 64 bit int
+ + + + + pointer pointer as an integer value
+ + + + - pointer-obj pointer from Tcl_Obj
+ + + - - pointer-utf8 pointer from String
+ + + - - pointer-utf16 pointer from Unicode
+ - - - - pointer-byte pointer from ByteArray
+ - - - - pointer-var pointer from ByteArray stored in variable. If the ByteArray is shared, then an unshared copy is made and stored back into the variable.
+ - - - - pointer-proc pointer to callback function constructed to call a Tcl proc.
+ + + + + struct structure aggregate

Installation

Download

The latest release can be downloaded from the following links:

The main repository can be accessed via web and git.

Build

Building on plaforms supported by TEA 3.2 should be painless.

Installation is like for any other TEA extension, minimally it consists of:

tar xzvf ffidl-0.8.tar.gz
cd ffidl-0.8
configure && make && make install

Custom configure options are implemented for selecting between libffi and libffcall (--with-libffi and --with-libffcall), for excluding callbacks (--disable-callbacks) and for enabling building of the ffidl test functions into the extension (--enable-test).

Custom libffi or libffcall

Ffidl no longer bundles libffi nor libffcall. Instead, the build system relies on pkgconfig to find the needed objects. If either library is installed in a non-standard location, use PKG_CONFIG_PATH to point Ffidl's build system to its location:

# Build libffi
tar xzvf libffi-3.2.1.tar.gz
cd libffi-3.2.1
configure --prefix=/some/path && make && make install
cd ..
# Build Ffidl
tar xzvf ffidl-0.8.tar.gz
cd ffidl-0.8
configure PKG_CONFIG_PATH=/some/path/lib/pkgconfig && make && make install

Building on Windows

The recommended setup for building for Windows is using Cygwing and MinGW. Assuming Cygwin is properly installed, the following commands on a Cygwin terminal build Ffidl for Win64:

setup-x86_64.exe -q -P make,mingw64-x86_64-gcc-core,mingw64-x86_64-tcl,mingw64-x86_64-libffi
# Build Ffidl
tar xzvf ffidl-0.8.tar.gz
cd ffidl-0.8
./configure \
    CC=x86_64-w64-mingw32-gcc.exe \
    CYGPATH='cygpath -m' \
    --enable-symbols \
    --with-tcl=/usr/x86_64-w64-mingw32/sys-root/mingw/lib \
    --prefix=/some/path \
    --exec-prefix=/some/path \
    PKG_CONFIG_PATH=/usr/x86_64-w64-mingw32/sys-root/mingw/lib/pkgconfig

Additional notes

On non-Darwin Unix and on Windows you should edit library/ffidlrt.tcl and look at the table of libraries in ::ffidlrt::libs and the table of types in ::ffidlrt::types. Either or both of these tables will probably need attention if you go further. You may need, for instance, to find, build, or install libraries for gmp and gdbm, or to adjust the pathnames for libc, libm, tcl, and tk.

Running

make test

will run the Ffidl tests (when configured with --enable-test). It is not quite a systematic test suite, but it does exercise a good deal of Ffidl's capabilities, and it has turned up some problems with libffi.

Demos

The demos directory contains several small and medium size examples of Ffidl bindings to shared libraries, and some code for making comparisons to other ways of doing the same thing.

atol.tcl a Ffidl binding to atol() and congeners
ffidlrt.tcl run time support for Ffidl bindings
gdbm.tcl a Ffidl binding to gdbm-1.8
getrusage.tcl a Ffidl binding to getrusage()
libm.tcl a Ffidl binding to libm
qsort.tcl a Ffidl binding to qsort
tkphoto.tcl a Ffidl binding to the Tk photo image.
pkgIndex.tcl hand built package index
gmp.tcl a Ffidl binding to gmp-2.0.2
gmpz.tcl arbitrary precision integers via gmp.tcl
gmpq.tcl arbitrary precision rationals via gmp.tcl
gmpf.tcl arbitrary precision floats via gmp.tcl
test-gdbm-1.tcl a test of the gdbm binding
test-gdbm-2.tcl a test of the gdbm binding
test-gmpz.tcl a test of the gmpz routines
time-libm.tcl a timing comparison of Ffidl and expr

library/ffidlrt.tcl will need attention unless you're running on Darwin. There are two functions, ::ffidl::find-lib and ::ffidl::find-type, which abstract library names and system typedefs out of the rest of the code. However, the abstraction on Unix is currently limited to the correct results for my Linux box. You'll need to rewrite the mapping for your own machine.

ibrary/ffidlrt.tcl also contains some examples of binding into the Tcl core itself.

demos/tkphoto.tcl allows extraction and insertion of photo image pixels as binary data. See tests/tkphoto.test for an example.

The demos/other/gdbm.tcl extension should be plug compatible with Tclgdbm0.6, a C coded Tcl extension for manipulating gdbm files. Since gdbm passes and returns structures, it also tests the Ffidl struct code.

The demos/other/gmp*.tcl extensions make a nice example. The main Gmp package wraps all the exported mpz_*, mpq_*, and mpf_* entries from the Gnu multiple precision library. The subsidiary Gmp[zqf] packages use the Gmp package to define arbitrary precision integers, rationals, and floats which are represented as strings. This isn't the most efficient way to do arbitrary precision arithmetic, but it is convenient, it does avoid needing to know what type mp_limb_t and mp_size_t actually are, and it does show how to use the underlying library if you want to build something more efficient.

Performance

Performance appears to be excellent, but I can't take any credit because libffi is doing most of the work. The demos/mathswig/time-libm.tcl script compares ::ffidl::callout wrapped libm functions to the Tcl expr versions of the same functions. You'll need to manually build the libmathswig0.5 dynamic library to provide a SWIG wrapped libm for comparison purposes. If you're running on Linux-x86 or Windows you can install Robin Becker's ::dll for another data point. demos/mathswig/time-libm.tcl will time them on the same functions.

Open Issues

A few issues have been closed since the initial release.

The Windows port is done.

Callbacks are implemented for the x86 architecture.

But there are many open issues.

Finding the right library is a pain. dlopen("libm.so") finds libm on my machine, but dlopen("libc.so") returns an error string decorated with binary characters while dlopen("libc.so.6") works. If you work with shared libraries you build yourself, it's not an issue, but for all the standard stuff there is no standard. In demos/ffidlrt.tcl the ::ffidl::find-lib function provides an abstraction for at least removing these issues one layer away from your Ffidl bindings to the library, but the implementation of the abstraction hasn't gone farther than listing where I find my standard libraries.

Discovering what type a type is is also a pain. Include headers are typically so heavily conditionalized, that one needs to search and search to find which typedef is actually implemented. In demos/ffidlrt.tcl the ::ffidl::find-type function abstracts these issues out of the Ffidl bindings, but again the implementation of the abstraction will need some work.

A backend for SWIG which generates Ffidl bindings might be useful.

There are some more pointer types which ought to be defined: a variant of pointer-var which requires an unshared value; a pointer to a native character string -- but couldn't that be pushed back to the Tcl layer?

Writing Tcl extensions with Ffidl is very much like writing C code in Tcl. I'm not sure what the actual required skill set is. But if you're not sure what you're doing, you might be in over your head. In any case, try not to take the core dumps personally.

Loading snippets of code into a Tcl interpreter with Ffidl loaded could be very hazardous, as in downloading "Try ME!" scripts from the web. There is no Ffidl_SafeInit(), we'd probably need signed scripts to even begin to consider such a thing.

I've looked at SWIG and at dll and seen that they very carefully duplicate any shared Tcl_Obj before attempting a conversion to Int or Double. I've also looked at the source for Tcl's expr command, and it converts objects to Double or Int and only duplicates shared objects when it finds a valid Int or a Double with an existing string representation. Ffidl only duplicates shared objects when processing pointer-var, though I'm open to explanations why it should do otherwise. It seems that if you pass a parameter to a typed function that you shouldn't be upset if the parameter is converted to that type.

Hmm, this is a really pared to the bone. It would be nice for newbies and experimenters and the careless if Ffidl implemented a debugging mode which verified that constraints were observed: 1) that Tcl_Obj string reps were not modified, 2) that Tcl_Obj bytearray reps were not modified outside their allocated sizes, and so on. This could be done by switching in an alternate implementation of tcl_ffidl_call() which made copies and verified the constraints after the call.

Some naming consistency in the demos. I seem to be reinventing my Ffidl extension style each time I start a new example.

Some style consistency in the tests. The tests just run, some generate descriptions, some report what they've done, some say nothing, some give summaries.

Credits

Robin Becker's ::dll package, which does much the same thing as Ffidl, provided the immediate inspiration for this work and pointed to the solution of some of the design issues for me. And Robin hisself has been very helpful.

Anthony Green's libffi package provided most of the initial implementation of Ffidl.

Bruno Haible's libffcall package for the clisp system provided an alternate implementation and a truly amazing example of cpp macrology.

Copyright, License, & No Warranty

Ffidl Version 0.8b0
Copyright © 2015-2018 Patzschke + Rasp Software GmbH, Wiesbaden;
Copyright © 2005 by Daniel A. Steffen;
Copyright © 1999 by Roger E Critchlow Jr, Santa Fe, NM, USA.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the ``Software''), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED ``AS IS'', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL ROGER E CRITCHLOW JR BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.