Grayspace is the codename for web-based operating system based on Hexarc.

Goal & Principles

The ultimate goal of Grayspace is to replace AI2 and become the primary interface for controlling and building on top of Hexarc. It should ultimately be a full-fledged operating system.

  • Think about this as the web version of Raspberry Pi. Or as Processing at Internet scale. Like Raspberry Pi, it should be very easy to get started and yet scale to real programs. Like Processing, it should focus (at least at first) on graphics, because they are the most flashy. But it should all be completely web based:
    • No special download; works on any browser.
    • Trivial to share stuff (but appropriate permissions enforced).
    • Easy access to all web services.
  • This should be a superset of Luminous. In other words, a program in Grayspace should have access to the Luminous engine and be able to do everything (including using other programs as input).
  • It should support everything that makes the "cloud" special, including the ability to run computation across thousands of machines.

Key Scenarios

  • Capture & Analyze: Imagine you there is a service (like ClearSkyClock) which provides some kind of daily data. You want to download this data periodically, store it somewhere, and then run analysis over it.
  • Graphics Rendering: All of the rendering tasks done by Luminous should be supported here. The advantage is that you can use multiple machines to render.
  • Hexarc Dashboard: It would be great to create a dashboards that displays Hexarc health: number of connections, CPU usage, disk space remaining, error logs, etc.
  • Transcendence Gameplay Analysis: Assuming there is a dataset of gameplay data stored on the server, we should be able to write various programs to analyze the data. For example, we could graph, pivot, filter appropriately.
  • Simulations: Imagine a gravity program simulation, but with access to multiple machines so we can handle thousands or millions of particles.
  • Games: One of the primary uses (as a hobbyist tool) should be to write games (particularly multiplayer).

Core Concepts

Programs

  • We distinguish between a source file, which contains the code for a program, and a running program.
  • Source files are kept in a standard (per user) file system in Hexarc. The user manages these files like any other data file. There is nothing special about them.
  • At any time, a user may run a source file to create a running program.
  • The program gets a unique URL, generally relative to some user namespace. For example: hexarc.com/u/username/p/programname/instance.
  • The program uses a copy of the source code at the time of instantiation. If the source file is changed later, the running program is not affected, until the user halts and re-runs it.
  • The user may run as many instances of a given source file as they want (each will get a unique URL). There may be different versions of a source file running concurrently.
  • A program has a specific interface which defines how it interacts with the world (and the user). For example, one interface type is console. A console program defines a single entry-point which takes a string and returns a string.
  • The entry-point is executed by the Hexe Engine, which tracks compute cycles (and assigns them to a user), and caches processes.
  • Display code in JavaScript handles user input/output to the console program.
  • A program is stored as an entry in the database (AeonDb), which tracks the state of the program persistently.

Program Example

  1. A user runs a console program, defined in a source file. The source file has two entry-points: OnExecute and OnTextInput.
  2. We assign the program a unique URL in the user's namespace. E.g., hexarc.com/u/george%20moromisato/p/example/114.
  3. We create a program entry in the AI3.programs table.
  4. We navigate to the URL, which shows a basic console interface.
  5. We call OnExecute (without parameters) and get back 0 or more lines of text. We display that text in the console.
  6. The call to OnExecute is handled by the Hexe Engine, which gets the program code and the username. The Hexe Engine tracks compute cycles used by a user and throttles back if the user has used too many cycles (and others are waiting).
  7. We should probably send partial results back to the caller, so that we don't have to wait for the whole computation. This also allows for infinite loop programs, which need to be terminated by the user.
  8. Note that we persist the console buffer in the program entry (or on some other table). Though we may need to access pieces of it for performance. [This counts towards the user's storage quota. Also, perhaps the buffer is stored in the user's data directory.]
  9. Now the user enters some text input. We call OnTextInput and get back some result.
  10. Note that the Hexe Engine should cache the code and process so that we don't have to send the code down for every call to OnTextInput. The Hexe Engine should be free to discard its cache, in which case it would return a message telling the display to resend the code.

Ideas and Questions

User Machines

It should be possible for a user to add their own machines to the arcology. These machines would be tagged as belonging to the given user and would never get any tasks for different users (thus minimizing security issues).

From the perspective of the arcology, these machines would be untrusted to do anything except to handle a well-defined set of messages initiated by the user. For example, we would handle compute tasks for the user.

We might also cache user data on the user's machine for efficiency, but the truth would always be in the trusted part of the arcology.

The advantage of this system is that we could get the best of both worlds: fast computes and local storage for user tasks (great for Luminous) but global access to data.

Programs and Displays (Old)

  • One of the core concepts is a ''program''. A program is a user-visible execution process with an adaptable interface and a well-defined lifetime.
  • Users create programs with simple HexeLisp-based definitions (though we plan for other formats and languages).
  • It's probably important that programs can be defined entirely in HexeLisp, so that we can manipulate programs as data (introspect, generate code, etc.).
  • Programs can be ''instantiated'' into one or more running ''processes'' by the user (or, perhaps, by another program with user permission).
  • Users should be able to leave programs running, essentially forever. Of course, programs running in the background (without user interaction) might have different resource constraints.
  • Programs generally adapt to the display window they've been given, from full screen down to a small tile.
  • Every running instance of a program should have an URL, which persists for as long as the program does.
  • Programs have well-defined input and output streams, allowing the user to chain together two or more programs. This allows both for pipe-semantics and UI composition (e.g., list/detail).
  • Program windows can be arranged on ''display'', which is a virtual desktop occupying an entire screen. The user may create any number of displays, and all are persisted automatically.
  • NOTE: The key problem will be the separation of computation (which happens in HexeLisp and on the server) from the display (which is likely HTML5 canvas). We don't want people to have to learn HTML5, but we can't run HexeLisp on the client. We need to come up with simple abstractions to generate all appropriate graphics on HTML5. Or we need to cross-compile HexeLisp to JavaScript (when appropriate). Or we need to render frames on the server and stream them up to the client (maybe even video-encoded). Most likely we'll need to use all three techniques. We can start with something simple. Imagine CURSES, but for well-defined GUIs. At minimum we can start with a character-mode abstraction and add more abstractions later. If we can implement Rogue (or something), that would be good milestone.

Data (Old)

  • How is data managed? Ideally, we should have a standard interface for a program to access many different kinds of data, including table data and blob/file data.
  • Every piece of data accessible to a program should have a unique and uniform name.
  • There are multiple storage systems, some defined by Archons (e.g., Anacreon or Luminous), but they should all have a reasonably consistent interface.
  • We need to support external systems via URLs.
  • There is a built-in storage system, which stores all user data (including programs). This system should handle file-scale objects (and it should automatically use file storage for large items).
  • For now we require programs to load their entire data in memory. For more complex problems with large data, we rely on separate interfaces (e.g., Luminous might expose special interfaces).
  • We need to ensure proper sandboxing of programs. We should not allow a shared program to have access to a different user's data without explicit permission.
  • The built-in storage system is a simple (flat, but tagged) collection of data+metadata objects. The metadata is well-defined and includes a type (e.g., file, program, JPEG, etc.). The user can define custom types by defining a program that can handle the type.