Archive for June, 2010

How to Keep Physics Engine Load Low

What is ODE?

Today most OpenSim installations use Open Dynamics Engine “ODE” as physics engine. It is delivered as integrated part of current OpenSim versions. It is used to simulate a (more or less) realistic physical environment in the virtual world. The physical behavior of physical objects, like some vehicle types, and avatars is calculated by ODE.

Under certain circumstances the physics engine can be very busy and that physics engine load can be experiences as server lag. This article tries to describe how to identify physics engine lag and how to improve your regions to cause less such lag.

Additionally to the server lag you experience much physics engine activity might cause that OpenSim consumes more and more memory over time, due to memory leaks. This may cause lower stability and more frequent restarts of your OpenSim installation.

How to identify physics engine lag?

Enter “show stats” on your OpenSim console window. After that you will see some output like this:

Asset cache contains        0 assets
Latest asset request time after cache miss: 0s
Blocked client requests for missing textures: 0
Asset service request failures: 0

Abnormal client thread terminations: 0

Initial inventory caching failures: 0

Dilatn  SimFPS  PhyFPS  AgntUp  RootAg  ChldAg  Prims   AtvPrm  AtvScr  ScrLPS
  1.00      55    45.5     0.0       0       0    3516       0     419      14

PktsIn  PktOut  PendDl  PendUl  UnackB  TotlFt  NetFt   PhysFt  OthrFt  AgntFt  ImgsFt
    27      10       0       0       0     5.3     0.0     4.3     0.0     0.0     0.0

Allocated to OpenSim : 152 MB

Have a look at the value shown under the label PhysFt. This number is the so called Physics Frame Time. If that value is high, above 20 or even higher,  your physics engine is quite busy. Best run “show stats” multiple times and check if you always get more or less the same PhysFt values or if you see peaks of physical engine activity.

If you see such peaks, even if nobody is currently visiting your region (use “show users” command on the OpenSim console to check that), then you probably have moving objects causing physics engine lag. Otherwise it is possible that moving avatars cause such lag, for example by walking on prim surfaces or by colliding with other objects or avatars.

The PhyFPS value shows the Physics Engine Frame Rate. Values above 45 fps are good. Usually high PhysFt values cause low PhyFPS frame rate values.

You can also display the Physics Engine Frame Rate in the viewer. Select the menu item “View > Statistics Bar”. The Statistics windows opens and shows the Physics FPS value under the label Simulator. If you cannot see it, simply click on the label Simulator.

Reasons for physics engine lag

In the following situations the physics engine has to do calculations:

  • Movements of physical objects
    • unscripted physical objects (i.e. a ball)
    • scripted physical objects (i.e. vehicles implemented as physical objects)
  • Movements of avatars
    • avatar walking over terrain
    • avatar walking over a prim surface (normal prims or sculpties)
  • Collisions between non phantom objects and avatars
    • avatars colliding with each other
    • non phantom objects colliding
    • avatar colliding with a non phantom object

Many physical movements require the detection of collisions. Collisions are detected based on so called Bounding Boxes, rectangular boxes using the X, Y and Z axis and that contain the whole object, avatar or segment of terrain.

You can toggle displaying these internal Bounding Boxes in the viewer using the following menu item in the Advanced menu (which can you open using ctrl alt D):  Advanced > Rendering > Info Displays > BBoxes

Bounding Boxes

How to reduce physics engine lag?

To reduce physics engine lag you should do the following:

  • Try to avoid physical objects (i.e. use vehicles that are not physical objects)
  • Never create huge physical objects, because they easily cause many collision events
  • Make objects phantom if possible (collision detection ignores phantom objects)
  • Make moving/rotating objects phantom
  • Use llTargetOmega and llSetTextureAnim for viewer side rotations/animations
  • Do not use hollow prims with the intention to let other objects or avatars move inside
  • Do not use too complex, rough sculpty prims and terrains as walking surface for avatars

The general rule is to try to avoid collisions of Bounding Boxes as much as possible to reduce physics engine load.

By the way, that is also the reason why avatar attachments are always phantom objects.

One of my renters could reduce Physics Engine Frame Rate values from about 50 with peaks up to 150 to much lower values of about 4. The main reason was a hollow prim with a scripted vehicle constantly moving inside. This did cause physics engine lag. But a bigger issue was the constantly growing memory consumption (memory leak) and thus more frequent region restarts than necessary.


This is a short description of the more cryptic labels of the “show stats” console output:

  • Dilatn time dilation
  • SimFPS sim FPS
  • PhyFPS physics FPS
  • AgntUp # of agent updates
  • RootAg # of root agents
  • ChldAg # of child agents
  • Prims # of total prims
  • AtvPrm # of active prims
  • AtvScr # of active scripts
  • ScrLPS # of script lines per second
  • PktsIn # of in packets per second
  • PktOut # of out packets per second
  • PendDl # of pending downloads
  • PendUl # of pending uploads
  • UnackB # of unacknowledged bytes
  • TotlFt total frame time
  • NetFt net frame time
  • PhysFt physics frame time
  • OthrFt other frame time
  • AgntFt agent frame time
  • ImgsFt image frame time

Snoopy Pfeffer
Founder and CEO of Dreamland Metaverse

Pro and Cons of Virtual Private Servers (VPS) for OpenSim

When you have a look at the OpenSim hosting market today, you quickly recognize two kinds of hosting providers. One group of providers offers OpenSim based on dedicated server hardware, while the other group of providers offer OpenSim on so called Virtual Private Servers (VPS), where many such virtual servers run on a single hardware server. The second group of providers mainly tries to sell OpenSim through low prices.

The question is, if Virtual Private Servers (VPS) can be a cheap alternative to dedicated servers for OpenSim or in which situations it might be an alternative.

Virtual Private Servers

VPS is a technology that was developed for cheap web hosting by being able to share expensive hardware resources by as many web servers are possible. Compared to basic web hosting services, it additionally allows the web site provider to use some advanced web hosting features, like server side scripts or to use of database storage. This are useful features for better looking and more feature rich web sites.

VPS bases on the assumption that web sites of smaller and medium sized institutions usually do not get traffic most of the time. During these times the processes of such web sites can be put asleep and the data is temporarily
offloaded on slow hard drives to free main memory for currently active web sites. This allows many web sites to share the same physical resources (CPU, memory, bandwidth), which keeps costs low.

VPS providers apply a so called Fair Use Principle. If one hosting customer uses too many ressources (CPU, memory or bandwidth) they take measures to ensure that the overuse of one customer does not affect the service quality of other customers. This can mean that the available bandwidth of that customer is cut back or that the processing power is reduced to ensure nobody else is affected negatively by that overuse.

Now the question is, if VPS as suitable for OpenSim or in which cases VPS might be an alternative to host OpenSim.

OpenSim processes, like processes of other real time applications like voice-over-ip or video conferencing never sleep like web server processes. Even if there is no visitor in an OpenSim region, a typical OpenSim process anyway constantly consumes between 0.5 and 3% processing power of a processor core.

This permanent use of processing power is untypical for web server applications VPS was mainly developed for. Depending on the kind of visualization software used, such processes are anyway put asleep from time to time, to avoid negative impact on other VPS instances, although these processes should be kept running. This is done to enforce the Fair Use Principly and to ensure that other VPS instances on the same hardware server get enough computing power and physical main memory.

As a consequence OpenSim users experience lag, because of regular storing and loading of data to slow hard drives and because of the overhead and delays of constantly having to start and stop processes.

These negative effects increases the more users are on one hardware server providing VPS services. And it gets even worse if there are more such processes overusing resources on the same server hardware. Such processes with permanent resource demands begin to fight for the sparse hardware resources, because the basic assumption that most (web server) processes can sleep and offload data most of the time is not true anymore.

Bandwidth can be another issue, because OpenSim creates peaks of data traffic. Depending on the virtualization technology used, this also causes issues, if the peak bandwidth is limited. That can cause crashes and instabilities during logins and teleports.

Because VPS tries to store data on hard drives as often as possible to safe main memory, the additional time needed to load data from that slow storage cause additional lag. The probability of such delays increases the more memory an OpenSim process uses. Bigger regions with many prims, scripts and visitors quickly become unusable because of the permanent swapping or paging of memory needed by these OpenSim process.

This effect also increases the probability that Mono threads block or crash completely, if OpenSim runs on top of Mono. This causes instabilities and more frequent crashes of OpenSim regions.

What a VPS Hosting Company Says

I had a long discussion with the CTO of a big VPS hosting company and he did clearly say: “VPS is not suitable for any real time application [voice, video streaming or virtual worlds].”

He said once they did host a voice conference on a VPS server for a customer, but that this was just done for a marketing show. In reality that customer was the only one on the hardware server used, to ensure high service quality. So in fact that customer did use a dedicated server for that marketing campaign, although it was said that it’s a virtual server.


VPS is not suitable for regions with much contents (prims and scripts) or if many visitors need to be supported. Beside that the service quality is substantially lower: Visitors experience more lag, less stability and more frequent region crashes.

On the other hand, if someone just needs a small region with not too many prims and scripts and if just 2 or maximum 3 visitors are enough, then VPS offers a cheap alternative for beginners. But most region owners will quickly reach the limit and will want to be able to add more contents to their region, without having to experience a quickly degrading service quality.

For professional users, like corporations, shop owners, dance club owners, etc. VPS is definitive no alternative for professional OpenSim hosting using dedicated server hardware.

Dedicated Servers

Dedicated servers are the best choice for bigger regions and regions that have to support many visitors.

The prices of OpenSim regions hosted on dedicated servers very much depend on the processing power, main memory and network bandwidth provided. Good offers for beginners start with 512 MB main memory per region, while the typical average users need up to 1 GB. High end users that use many prims, scripts and have many visitors need up to 2 GB of memory for one region.

General Advices

In any case be careful if an OpenSim hosting providers promises you very high numbers of prims, scripts and visitors, especially if no information about the underlying hardware platform is given or if you see that the amount of memory, the included network bandwidth or processor seem not to be sufficient for what they promise.

At the end it only counts what you get and not what they did promise you, despite how low their price is!

You should also carefully check what is included in the hosting service package. Do they ensure high availability with professional 7×24 service monitoring? What kind of web based or in-world management tools are provided? What about backups? Are optional features like voice, search, groups, off-line messages, money, HyperGrid and mega regions included?

Data backups are a must and should include daily backups and on demand backups to protect you from data loss. Database backups are the most important, because they store all important data, but customers should also be able to get backups of their region contents as OAR archive files.

Customers should also get web based and/or in-world tools to manage their regions (display status, restart, etc.) and in-world monitoring panels for regions.

Finally and most important check if they have good references for their hosting services and for customer service? Service quality very much depends on how professionally OpenSim is managed and run. Good references are an indication if a hosting provider does his job well.

Currently many OpenSim hosting companies out there have problems to deliver what the promise. With good references the chances are higher that you will be happy customer, using good OpenSim hosting services for a reasonable price.

Snoopy Pfeffer

CEO & Founder of Dreamland Metaverse

Blog Stats

  • 21,588 hits


June 2010
« Dec