Adam starts to cover monthly what has happened at Opensim. Read on if you want to find out what are the highlights of the month! Groups support is progressing which is a feature waited by many.
With May coming to a close, I would like to start the first of a
monthly feature - some explanations of what's going on under the
surface of OpenSim. But wait, you may ask; doesn't
Justin already provide excellent weekly reports on what's
been committed? Of course - where this differs is in the level of
detail, particularly on infrastructure projects. Justin does a
weekly feature which means he only has time and space to report on
features in bullet form. Where I plan to differentiate with these
posts is in the details - not only what has been committed, but
also where it is going.
There has been a lot of progress this month - starting at
revision 9358 there have been several major areas of focus.
Networking: LLUDP Improvements
We now support the Multiple Object Update packet format for
Linden UDP clients (most notably Second Life & realXtend), this
builds on work that Mikko of realXtend provided late in April that
did the same for Avatar Updates. The work done by Melanie reduces
the total number of packets that need to be tracked by packing
multiple updates into each packet - resulting in a significantly
lower workload for our packeting code. The Avatar packet
aggregation with 100 Avatars in the region results in a massive 71%
reduction in the number of packets transmitted, and a 29% drop in
CPU requirements, according to
tests done by Intel for ScienceSim. Object aggregation
produces similar results (although hasn't been measured
precisely.)
Packet Pooling - the next major improvement is in 'packet
pooling' - one of the drawbacks to managed languages is very little
control over optimising garbage collection. OpenSim handles
potentially thousands of packets each second, in our previous
packet system for each of these packets, we created several classes
representing the internal data structures. (eg, Byte[] Packet
becomes MyPacketObject) - while the processing is fairly quick,
creating these packets results in the Garbage Collector handling
tens of thousands of tiny unused classes at each collection. On
.NET this runs so-so, but on Mono it runs atrociously.
Packet Pooling solves this problem by recycling classes - rather
than leaving a dead class to be garbage collected, we stuff new
data into it and re-use the class instance. We previously did this
work back in 2007 with the help of 3Di developers, however it was
at the time proven unworkable due to problems with old data
persisting into new data blocks. Melanie managed to solve this
issue, and nearly two years later we have it re-enabled. Early
pooling code did not show much promise past 40 avatars (although up
to that point was fairly dramatically better) - however more
profiling done by intel managed to locate the problem as
being due to a finite sized pool - a patch fixed this problem and
the overall improvement was a further 40% reduction in CPU
requirements for 100 avatars (total reduction for this and
aggregation is a whopping 57% reduction in CPU requirements for a
busy sim).
Overall, the OpenSim networking code has been improved very
significantly (especially with April's texture resend fixes), and I
would appreciate seeing more testing with large userbases - the
Wright Plaza weekly meetings have been running better than they
have for a long time, but only represent a peak of 35 concurrent
users.
Some early bugs did present themselves with this networking code
- most notably running into several MTU limits for international
users (apparently most international trunk lines will simply drop
rather than fragment big UDP packets), myself, Nebadon and Melanie
managed to identify and fix this problem.
I introduced, then disabled support for NAT Loopback - this
allows you to emulate a loop-back compatible router for those
hosting from home and unable to get a public internet IP. This is
currently commented out due to issues with compatibility in Grid
Mode (however Standalone mode can be made to work effectively) -
further improvements will be in the realm of adding UPnP support
allowing you to host a region from home reliably. Finally I also
introduced the capability to use IP and DNS based bans (including
wildcards), Dr Scofield improved this with some stability
checks.
Groups Support
mcortez has been working on adding a fully functional
implementation of groups into OpenSim this month - previously
located as part of the flotsam project, the groups implementation
has hit core. Early revisions ran into a problem with a permissions
exploit and group deeding (allowing anonymous users to group deed a
parcel), however these have been fixed and it is ready for further
testing. My initial feedback is that it looks good - things such as
group chat are behaving well and responsively, although it is not
yet fully functionally complete yet (particularly around the area
of group permissions).
New Region Modules Framework
I'm not entirely sure if this was introduced this month -
however a large amount of work was done porting from IRegionModule
to the new ISharedRegionModule interface[s]. This new version is
syntactically nearly identical, however has better support for
removing regions at runtime, and also utilizes the Mono.Addins
framework, which will be essential as we begin to add
PECL/CPAN-style module repositories to OpenSim (although exactly
when this happens is an unknown). The older region module format is
going to remain working for quite some time yet - however if you
are feeling adventurous you can investigate the new format and look
at porting modules across.
New Services Framework
As part of a general improvement of OpenSim code quality, Diva
and Melanie improved the services framework this month, deleting
the old OGS1 'UGAIM' in favor of a new 'OpenSim.Services' binary -
this also means that the Grid mode and Standalone mode now share
all the major infrastructure internally (Standalone runs the grid
services in 'private' now) - this reduces code complexity and makes
it easier for us to add and adjust the services framework.
This framework has also made it easier to edit and adjust module
points for Grid Services, meaning it is now possible to write Grid
Service plugins - such as asset caches, external login systems,
etc. Diva has been utilizing these to improve the functionality and
quality of the Hypergrid technology. A new cache was implemented
using GlynnTucker cache - after it was discovered the previous
asset cache did not infact, cache anything, early testing indicates
this one works.
Finally, Arthur implemented some improvements to the User
Service framework which prevents people from seeing the 'You are
already logged in' message' in grid mode.
Remote Control, Administration and Web Integration
BlueWall introduced and improved his JSON Stats plugin system
for OpenSim which allows remote monitoring via AJAX applications of
a region. Melanie introduced a 'remote console' system that
utilizes REST, and I introduced some basic capacity for Grid Owners
to be able to send grid-wide announcements for maintainence
purposes. Dr Scofield improved the XMLRPC-based Remote Admin module
to handle additional formats. All of these will no doubt go towards
improving administrator remote tools and web interfaces.
Inventory Archives
Justin continued work on Inventory Archives - these allow you to
save a copy of your entire OpenSim inventory to a file on disk,
which you can then restore on any other compatible OpenSimulator
instance. For those who work on multiple seperate OpenSim instances
(such as a personal grid, public grid, etc) being able to
synchronise your inventory between the two is a big help. IAR
archives are still highly experimental and should not be relied on
yet for any production purposes.
Justin also added the ability to store creator IDs as a URL
rather than a UUID - this allows you to reference foreign grid
avatars as a creator.
Prim Meshing
Dahlia continued work on the internal 'Prim Mesher' used for
physics. Improving sculpty physics shells to be more accurate and
support properties such as Mirror, improving the overall meshing
performance and adding the capability to disk-cache converted
meshes for faster startups.
XML Serialisation Formats and Region Archives
Sean and Justin made several improvements to our XML
Serialisation support including adding unit tests to the
serialisers. This will improve overall reliability and support for
OAR files as we move forward (preventing OARs from becoming
version-incompatible with each other.), OARs were also improved
with support for a wider variety of .tar formats courtesy of Dr
Scofield. Finally, OARs were renamed from .tar.gz to .oar as the
default filename.
Scripting
I improved MRM scripting, both the Engine and the API. The
engine has been improved to support real-time loading of external
binaries, including debugger support. This when compiled with
the
MRM Loader project (located on the forge) allows you to
write (and debug!) MRM scripts from within Visual Studio
(very cool.), MRMs were also improved with additional
security checks, fixing of certain properties (such as
Avatar.Position), events (World.OnNewUser) and introduction of an
Audio API (World.Audio and IObject.Sound).
LSL was improved with support for the LSL Webserver API going
into production on the Second Life(tm) Beta Grid. Also implemented
was llAttachToAvatar, llDetachFromAvatar and a host of other
improvements.
IRC Clientstack
As mentioned earlier, OpenSim now has a ClientView which
supports a primitive text-only IRC protocol support. This allows
you to connect to a region for chat facilities using only an IRC
client - this may be an advantage over the previous IRC Bridge
module where IRC users were not represented inworld with an
avatar.
Final Words
If I missed anything - let me know, Justin's notes are likely to
be slightly more comprehensive on smaller non-architectural
changes.