NetBeans as a Tools Platform - DRAFT

Version: April 5, 2006
Author: Jesse Glick

These are some introductory notes about use of NetBeans APIs and code as the basis for a "tools platform".

See also:

Yarda's Platform Road Map

Quick HOW-TO on using the NetBeans core and accompanying source code

Third-party applications built using NetBeans, some of which employ it as a platform rather than an IDE.

Issue #17815 tracks progress of "platformizing" the core

Basic Definitions

It is useful to start by giving some terms that could be used in discussion.

API, OpenAPIs

By the "OpenAPIs" is meant all the APIs that form the core of NetBeans: Java packages in org.openide.**, and such expected behaviors as are specified in the "NetBeans Open APIs" document set (normative documents only, not informational documents such as FAQs) as published on http://openide.netbeans.org/. By an "API" in the singular is meant one of the component APIs, which would typically cover one or more Java packages, and has a prose document describing its expected behavior: for example, the "FileSystems API" (org.openide.filesystems.* and openide/api/doc/org/openide/filesystems/doc-files/api.html) or the "DataSystems API" (three packages and a document) or the "Services API" (pieces of several packages and a document). Currently the listed APIs are:

• Modules
• Services
• FileSystems
• DataSystems
• Nodes
• Explorer
• Actions
• Compiler
• Execution
• Debugger
• Editor
• Window System
• Options
• XML
• Java Hierarchy
• Utilities

The Utilities API consists of a number of unrelated utility classes rather than a coherent whole. The Services API subsumes lookup and expected behaviors of layer installation (Modules API) and instance and XML files (DataSystems API). The XML API currently lacks a specific prose document. The Java Hierarchy API is expected to be deprecated in favor of the Metadata Repository in the future using a proposed Java metamodel.

SPI vs. client API

"Client" usage of an API means to call it externally and use the results: for example, obtaining a FileSystem and calling methods on it. "SPI" (Service Provider Interface") usage means to provide new objects (often using protected members and so on): for example, defining a new kind of FileSystem.

Core

The NetBeans core, in org.netbeans.core.** Java packages and with some additional infrastructure, implements some abstract objects in the APIs and provides additional functionality, including much of the basic application GUI. Its exact behavior is not documented except that it must help the APIs behave as they are documented to do, in some cases. Many APIs have a "core API" (as well as client and SPI) which provides integration points for the core to register its functionality with the APIs. In all cases these are designed to be flexible enough that an alternate core implementation could fulfill the same functions.

Levels of Reuse

Different uses of NetBeans as a "platform" reuse existing NetBeans IDE code to a varying degree. To some extent the boundaries between these levels are fuzzy.

Library reuse

By "library reuse" is meant use of only particular APIs from the available set in a completely different kind of application. For example, someone may wish to take advantage of the filesystem abstraction offered by the FileSystems API; in this case the developer would reuse the FileSystems API and also the Utilities API (which is needed by FileSystems for some purposes). The APIs would probably be incorporated into the new application as standalone JAR files.

Core reuse

By "core reuse" is meant an application which adopts the NetBeans core as well as the complete Open APIs, but which employs a quite different GUI. Certain IDE-specific parts of the core and APIs, such as the compilation engine, might be omitted or left unused. Basic GUI components such as the Explorer tree view and the window system would probably be used as is, but the contents of the application could be quite different: different menus, toolbars, workspace layouts, and so on. Application-specific functionality would be provided by non-NetBeans modules according to the APIs. Minor changes to Java sources in the core may be necessary in some circumstances.

GUI reuse

"GUI reuse" can be used to refer to an application which uses the NetBeans core and APIs essentially as is, including its standard GUI components and menu / toolbar / workspace arrangements, but with a different set of modules. Only infrastructural NetBeans modules (projects, autoupdate, etc.) are reused. This is suitable for applications which are essentially development tools, but do not significantly overlap the NetBeans IDE in scope. Minor GUI customizations such as main window title are likely, as well as whatever additions are made by modules.

IDE reuse

"IDE reuse" means keeping the NetBeans core, APIs, and many NetBeans modules intact, and adding some custom modules. The result is an IDE, possibly Java-oriented. Changes to the core would be limited to minor textual substitutions for branding purposes. The result is akin to an alternate distribution of NetBeans with a different focus: for example, focusing on a non-Java language, or on modeling.

Branding

"Branding" is described in technical detail at http://www.netbeans.org/i18n/, which describes how it is a special case of internationalization in the NetBeans framework. It permits an application built on the NetBeans APIs to specify a "branding" which, like a locale, lets certain resources be substituted at runtime according to the active branding name. Most things of interest in NetBeans pertaining to the GUI are now specified via "resource bundles" (*.properties) and "layers" (mf-layer.xml), as well as other data formats such as images. In most cases these resources can be given locale- and/or branding-sensitive "variants", of which the most specific will be selected at runtime. Furthermore the variants are contained in physically distinct JAR files, permitting them to be added without modifications to basic installation files. For example, the contents of the main menu bar in the IDE can be completely or incrementally customized by providing separate JAR files containing branded variants of the layers for the core and any relevant modules.

Compatibility

"Compatibility" of the APIs is described in several different meanings at http://openide.netbeans.org/nck/NCK_Terms.html. For our purposes, it can mean "functional compatibility", i.e. code written in compliance with the APIs as they are documented at one point in time, should continue to work within the behavioral limits described by the APIs at that time, when using a newer version of API code. For full functional compatibility to be realized in the case of library reuse, certain APIs must be implemented manually: for example, declarative MIME resolvers are specified by the FileSystems API but must be privately implemented somewhere, as the NetBeans core does.

Where Are We?

As of this writing, GUI and IDE reuse are fully possible and supported; core reuse is mostly supported but there are some caveats and unimplemented aspects; library reuse is supported only for certain APIs.

Standalone library availability

This list is probably not comprehensive, but should give a good picture of what is already usable in easily-digestible pieces.

Utilities API

The Utilities API can be packaged in a standalone library, openide-util.jar, with no runtime dependencies on other code. (There are some compile-time dependencies on other APIs from a few places which could not be removed for reasons of backwards compatibility; however if the other APIs are not present at runtime, unrelated parts of the API can be used without problems.)

To be complete, the system property org.openide.util.Lookup should be specified and give a class name implementing Lookup which will serve as the default lookup source in the system. If not supplied, the default lookup will contain no results for any query.

ant -f openide/build.xml lib-util

FileSystems API

The FileSystems API is available as a standalone openide-fs.jar, dependent on the Utilities API. Both SPI and client usages are supported. http://openide.netbeans.org/fs/ offers it for download with some instructions.

While mostly complete as it is, MIME resolvers specified in declarative XML format must be parsed and registered into the system if they are to be used. This is done in the core in the package org.netbeans.core.filesystems.*, though this implementation relies on other APIs (such as DataSystems) being available, as well as a system file system where such resolvers could be listed (Modules API). To use declarative resolvers without the full NetBeans core, an alternative parser and registration system would be needed. Otherwise, procedural resolvers can be registered and used so long as they are available from Lookup (see Utilities API).

ant -f openide/build.xml lib-fs

Nodes API

The Nodes API (including org.openide.cookies.* from the DataSystems API) can be made into a standalone openide-nodes.jar, depending on the Utilities API. Note that many cookies would make no sense without other APIs; however simple ones such as SaveCookie do. The Nodes API by itself permits nodes to be constructed and specified but gives no GUI for them.

ant -f openide/build.xml lib-nodes

Explorer API

The Explorer API (org.openide.explorer.** packages), depending on the Utilities and Nodes API as well as some generic actions from the Actions API and a few GUI components from the Utilities API (not otherwise packaged in openide-util.jar), can be used as a standalone openide-explorer.jar. In conjunction with Nodes it supports creating various GUI views such as trees or property sheets.

ant -f openide/build.xml lib-explorer

Modules API

The Modules API plus an implementation in the core is available as a standalone modules.jar: the non-deprecated parts of org.openide.modules.* together with selected classes from org.netbeans.core.modules.* can form an abstract module system which implements basic parts of the Modules API, with the help of the Utilities API: the notion of a module JAR with some attributes, collection of installed JARs, dependencies, module classloading, lookup of modules, and manifest registration. Recognition of specific kinds of manifest section could be added fairly easily without modifying existing code, and alternate implementations and usages of the layer registration system would be straightforward using the FileSystems API (which handles layer XML parsing and merging).

ant -f core/build.xml lib-modules

XML API

There is not a great deal of implementation in the XML API, and it would probably not be useful without other parts of the APIs. That said, it could be packaged as a standalone library depending on the Utilities API.

Wizards

This (currently uncategorized) mini-API could be packaged if desired: NotifyDescriptor, DialogDescriptor, and WizardDescriptor, depending on openide-util.jar. Some implementation to display the wizard dialog and buttons would need to be provided (the API code only controls the GUI of the dialog interior).

DataSystems API

Currently the DataSystems API is not at all separated from other APIs; it is heavily interlinked with all other APIs and cannot run without a significant amount of infrastructural implementation, such as is supplied by the core.

Compiler, Execution, Debugger APIs

These APIs are heavily tied to other APIs including DataSystems. It is not clear if they would have any value as standalone APIs.

Terminal emulator

The new terminal emulator intended as a replacement for the Output Window should be usable independently of the rest of the core. Currently it has not been packaged as such, but this would be relatively easy.

JavaHelp

As of this writing, JavaHelp support (required by the Open APIs) has been split into a separate module residing in core/javahelp/.

Editor

The editor module is split into a NetBeans-specific GUI, and a generic Swing-compliant text editor which can be used outside NetBeans entirely.

editor/demosrc/README.txt

Classfile parser

The Java bytecode reader used by sourceless Java classes is usable as a standalone library.

http://classfile.netbeans.org/

JavaCVS library

The Java-implemented native CVS client module is built on top of a generic standalone library which has been successfully used in an unrelated product.

ant -f javacvs/build.xml jars_independant

http://javacvs.netbeans.org/library/

Possibilities of core reuse

The NetBeans core could be reused with substantial GUI modifications if this were desirable. Things which could be customized using only branding (i.e. addition of branding variants of standard layers and other resources) include:

• most icons used by the core, as well the splash screen
• all human-visible text strings
• tips of the day (incl. contents, number, order)
• actions pool (if relevant)
• tasks in the upper part of the welcome dialog (if relevant)
• panels of setup wizard (if relevant)
• menu items in main window
• toolbars and toolbar items in main window
• list of workspaces, incl. initial windows and their positions and docking modes, titles, and toolbar configuration
• global keyboard shortcuts
• templates (if relevant)
• contents of options window (if relevant)
• available core filesystem types (default: directory and ZIP/JAR) (if relevant)

Items marked "(if relevant)" would not need to be customized at all in cases where they would never be presented to the user anyway, due to other changes. For example, if the menu item Tools | Options were suppressed and no direct equivalent given, then the GUI contents of the Options master node are irrelevant.

Some other things may be disabled using system properties or startup options:

• display of the splash screen (-nosplash)
• display of the setup and import wizards (netbeans.full.hack)
• display of the welcome dialog (netbeans.full.hack)
• display of any GUI at all (main window) (org.openide.TopManager)
• suppression of toolbar area and/or workspace switcher (netbeans.windows.small_main_window or netbeans.windows.nostatusline)

Planned but unimplemented:

• suppression of Processes node in Runtime tab (#19609)
• customization of initial lists of tabs in main explorer (Filesystems, Runtime, Workplace or active project); but main explorer can be suppressed from default workspaces (#19609 again)

Some other typical branding customizations possible not relating to the core but to common GUI-providing modules (e.g. utilities):

• list of bookmarks
• other URLs in the Help menu
• Update Center locations

Branding process

Branding of images and similar atomic resources permits the default image to be replaced with a branded variant. Branding of bundles permits individual keys to be overridden. Branding of layers permits individual entries ("files") to be added; removed; or modified (either in content or attributes).

The API Support module includes a facility for visually editing XML layers in the Explorer, and this includes an automatic variant generator: when the original base layer is available to the support, the branded variant can be edited either as it is (what will physically appear in the branded XML) or as what the desired merged result will be (where the branded XML is computed automatically as the difference from the requested result and the base layer).

Ease of maintainability of branding across releases of the unbranded original code varies according to what is being branded. Images and such oqaque resources tend to keep the same filenames for a long time, so this is not an issue. Bundles also tend to retain the same key names for long periods, so long as the basic form (number of subsitution parameters) and meaning are unchanged. In the case of layers, files do sometimes move from one folder to another or are renamed or refactored; the main practical limit on such movements is that many layer-supplied files are either in a fixed position (such as preloaded XML DTDs) or cannot be freely moved around without affecting compatibility of user settings (in the system/ directory of an IDE installation). If it is necessary to keep a layer entry in a stable position so as to simplify user upgrades from old IDE versions, typically the same consideration helps those who are branding the layer.

IDE and GUI reuse

These usage styles should be fully supported. Customizations to the core should not be needed beyond light text and image substitutions to match product branding conventions; all other functionality can be inserted with modules following the full set of OpenAPIs.

Licensing implications

Patches to existing files, both Java sources and other resources, must be released under the SPL license according to its terms, if the patches are used in a distributed product. Branding does not trigger this clause of the SPL since it involves only adding new files and not modifying existing ones.

What is Most Needed?

A number of things are missing from this picture from the standpoint of something using pieces of NetBeans.

Fuller documentation

The OpenAPIs official documentation, as well as other tutorials and whitepapers, focus on IDE reuse, and to some extent GUI reuse. Core reuse is barely covered in the documentation on branding; library reuse is essentially undocumented beyond what is specified in the OpenAPIs documentation, so developers would need to rely on the public mailing lists to get any questions answered.

There are currently no downloadable demos or sample applications demonstrating how to use individual libraries, or a branded core, in a different kind of program than the IDE.

There is currently no single web address or index of resources needed for partial reuse of NetBeans. There is one tutorial written by someone who did it, which lists a number of simple concrete steps that can kickstart the effort, and there is this document, and some information on the FileSystems API.

While the API Support includes a tool to help generate branded layers (such as may be used to customize the core), there is no tutorial demonstrating how to use it in a typical situation.

Tests

Unit tests using JUnit and the XTest framework do exist for many APIs and parts of the core. In some cases these cover standalone functionality of APIs that would be relevant for library reusers: for example, the FileSystems API and the core modules implementation both include unit tests which can be understood with the isolated library. Coverage of other APIs and potentially independent subsystems with unit tests is of variable quality. Clearly it is uncomfortable to use a library if a test suite demonstrating its desired behavior is not available: a test suite serves as a complement to documentation, as well as guaranteeing a certain level of quality and compatibility.

Furthermore, the independence of these subsystems is not mechanically tested at the moment. For example, the FileSystems API is supposed to be independent of other APIs other than Utilities at compile time as well as runtime. This convention is maintained by FileSystems API developers, but currently no automated test exists that would catch accidentally added dependencies on other APIs: the NetBeans daily build builds all APIs together, and even the creation of openide-fs.jar does not attempt a separated compilation. For example: a careless or uninformed developer could easily insert a call to TopManager.getDefault().getErrorManager(), rather than Lookup.getDefault().lookup(ErrorManager.class), into some class in org.openide.filesystems.*; the result would be that openide-fs.jar might at runtime throw errors, or if other APIs were available in the classpath, begin to load parts of the IDE core. (Note: as of this writing, ErrorManager.getDefault() is to be used instead, and the other alternatives are deprecated.)

Update: in the trunk there is a now a test suite which builds the packaged standalone libraries from org.openide.** packages and evaluates whether they are correctly packaged or contain illegal dependencies. Currently there are some failures in this test suite, but it is instructive for developers to run to ensure that new illegal dependencies are not added.

ant141 -f openide/test/stdalone-build.xml

Convenience builds and downloads

Currently only the standalone FileSystems API (and a fairly old version at that) is available for separate download; other standalone libraries can be built by fetching NetBeans sources and running special Ant targets (listed above under the API); and these do not necessarily build any accompanying documentation. This is unfriendly for someone who is interested only in getting and playing with a library, and not necessarily wish to download full sources and play with build scripts.

Also there is no official release schedule for standalone libraries, so an interested developer would need to guess at a good snapshot to take: current development sources? Sources from the last official NetBeans release? Since there is no quality control done on the standalone libraries, this would have to be a guess.

Greater decomposability

Some parts of the Open APIs have been isolated from the rest to the extent possible while preserving backwards compatibility for users of the complete APIs. For example, the FileSystems API is fairly well isolated; the Explorer API somewhat less so, for compatibility reasons: if freely refactored it could be made quite independent of all but Nodes and Utilities.

But other APIs, such as DataSystems, have not been tried yet. Anyone wishing to reuse such APIs will find it necessary to include essentially all of the OpenAPIs and a substantial amount of backing implementation, probably the full IDE core. Probably a complete separation of such an API from its nonessential dependencies would not be possible without significant and incompatible changes.

The core itself is hardly decomposed at all. Half of the module system can be used independently of anything but the Modules and Utilities APIs. Various BeanInfos may be used dependent only on what they describe. The terminal emulator is separate. There are a couple of non-GUI start classes which suppress display of any GUI and can be used for scripting or testing, and perhaps as a basis for a radically different GUI. Beyond that, it is a tangle of interdependent classes with little or no documentation; without serious development effort on the part of a potential user, it must be taken as a whole or not at all.

For the future it is thought that the core could be physically divided into a number of mutually independent (or carefully dependent) components, each implementing some API or major feature. Many such modules would be strictly nonvisual and noninteractive (for example the execution and security engine, or the innards of the module system); others would explicitly add GUI components (such as the window manager, or the settings GUI). This would open possibilities for developers to choose certain core modules to include in an application and not others. Since the core does not have a public API it is possible to refactor existing code without fear of breaking compatibility. See issue #17815 to track progress on this front.

One argument that the situation is not as difficult as it seems for reuse of the core is that many pieces of the core can simply be turned off or omitted with a few changes in Java sources. While true, this ignores maintenance: a manually customized or stripped-down library is difficult and dangerous to maintain over several releases of the canonical code. Most version-control systems have a mechanism for importing "vendor branches" of code while maintaining local customizations; after some manual merging, this still leaves the reliability of the resulting code a question, since the user is the first and only developer to see or run it. Branding and other forms of incremental declarative customization are preferable, though a little maintenance from release to release is still necessary. The best situation would be one in which each major piece of the core were really defined in an independent module, meaning whole sections of it might be disabled or replaced safely and easily.