How to Keep Physics Engine Load Low

What is ODE?

Today most OpenSim installations use Open Dynamics Engine “ODE” as physics engine. It is delivered as integrated part of current OpenSim versions. It is used to simulate a (more or less) realistic physical environment in the virtual world. The physical behavior of physical objects, like some vehicle types, and avatars is calculated by ODE.

Under certain circumstances the physics engine can be very busy and that physics engine load can be experiences as server lag. This article tries to describe how to identify physics engine lag and how to improve your regions to cause less such lag.

Additionally to the server lag you experience much physics engine activity might cause that OpenSim consumes more and more memory over time, due to memory leaks. This may cause lower stability and more frequent restarts of your OpenSim installation.

How to identify physics engine lag?

Enter “show stats” on your OpenSim console window. After that you will see some output like this:

ASSET STATISTICS
Asset cache contains        0 assets
Latest asset request time after cache miss: 0s
Blocked client requests for missing textures: 0
Asset service request failures: 0

CONNECTION STATISTICS
Abnormal client thread terminations: 0

INVENTORY STATISTICS
Initial inventory caching failures: 0

FRAME STATISTICS
Dilatn  SimFPS  PhyFPS  AgntUp  RootAg  ChldAg  Prims   AtvPrm  AtvScr  ScrLPS
  1.00      55    45.5     0.0       0       0    3516       0     419      14

PktsIn  PktOut  PendDl  PendUl  UnackB  TotlFt  NetFt   PhysFt  OthrFt  AgntFt  ImgsFt
    27      10       0       0       0     5.3     0.0     4.3     0.0     0.0     0.0

MEMORY STATISTICS
Allocated to OpenSim : 152 MB

Have a look at the value shown under the label PhysFt. This number is the so called Physics Frame Time. If that value is high, above 20 or even higher,  your physics engine is quite busy. Best run “show stats” multiple times and check if you always get more or less the same PhysFt values or if you see peaks of physical engine activity.

If you see such peaks, even if nobody is currently visiting your region (use “show users” command on the OpenSim console to check that), then you probably have moving objects causing physics engine lag. Otherwise it is possible that moving avatars cause such lag, for example by walking on prim surfaces or by colliding with other objects or avatars.

The PhyFPS value shows the Physics Engine Frame Rate. Values above 45 fps are good. Usually high PhysFt values cause low PhyFPS frame rate values.

You can also display the Physics Engine Frame Rate in the viewer. Select the menu item “View > Statistics Bar”. The Statistics windows opens and shows the Physics FPS value under the label Simulator. If you cannot see it, simply click on the label Simulator.

Reasons for physics engine lag

In the following situations the physics engine has to do calculations:

  • Movements of physical objects
    • unscripted physical objects (i.e. a ball)
    • scripted physical objects (i.e. vehicles implemented as physical objects)
  • Movements of avatars
    • avatar walking over terrain
    • avatar walking over a prim surface (normal prims or sculpties)
  • Collisions between non phantom objects and avatars
    • avatars colliding with each other
    • non phantom objects colliding
    • avatar colliding with a non phantom object

Many physical movements require the detection of collisions. Collisions are detected based on so called Bounding Boxes, rectangular boxes using the X, Y and Z axis and that contain the whole object, avatar or segment of terrain.

You can toggle displaying these internal Bounding Boxes in the viewer using the following menu item in the Advanced menu (which can you open using ctrl alt D):  Advanced > Rendering > Info Displays > BBoxes

Bounding Boxes

How to reduce physics engine lag?

To reduce physics engine lag you should do the following:

  • Try to avoid physical objects (i.e. use vehicles that are not physical objects)
  • Never create huge physical objects, because they easily cause many collision events
  • Make objects phantom if possible (collision detection ignores phantom objects)
  • Make moving/rotating objects phantom
  • Use llTargetOmega and llSetTextureAnim for viewer side rotations/animations
  • Do not use hollow prims with the intention to let other objects or avatars move inside
  • Do not use too complex, rough sculpty prims and terrains as walking surface for avatars

The general rule is to try to avoid collisions of Bounding Boxes as much as possible to reduce physics engine load.

By the way, that is also the reason why avatar attachments are always phantom objects.

One of my renters could reduce Physics Engine Frame Rate values from about 50 with peaks up to 150 to much lower values of about 4. The main reason was a hollow prim with a scripted vehicle constantly moving inside. This did cause physics engine lag. But a bigger issue was the constantly growing memory consumption (memory leak) and thus more frequent region restarts than necessary.

Appendix

This is a short description of the more cryptic labels of the “show stats” console output:

  • Dilatn time dilation
  • SimFPS sim FPS
  • PhyFPS physics FPS
  • AgntUp # of agent updates
  • RootAg # of root agents
  • ChldAg # of child agents
  • Prims # of total prims
  • AtvPrm # of active prims
  • AtvScr # of active scripts
  • ScrLPS # of script lines per second
  • PktsIn # of in packets per second
  • PktOut # of out packets per second
  • PendDl # of pending downloads
  • PendUl # of pending uploads
  • UnackB # of unacknowledged bytes
  • TotlFt total frame time
  • NetFt net frame time
  • PhysFt physics frame time
  • OthrFt other frame time
  • AgntFt agent frame time
  • ImgsFt image frame time

Snoopy Pfeffer
Founder and CEO of Dreamland Metaverse

snoopy.pfeffer@yahoo.com
http://www.3dmetaverse.com/

Pro and Cons of Virtual Private Servers (VPS) for OpenSim

When you have a look at the OpenSim hosting market today, you quickly recognize two kinds of hosting providers. One group of providers offers OpenSim based on dedicated server hardware, while the other group of providers offer OpenSim on so called Virtual Private Servers (VPS), where many such virtual servers run on a single hardware server. The second group of providers mainly tries to sell OpenSim through low prices.

The question is, if Virtual Private Servers (VPS) can be a cheap alternative to dedicated servers for OpenSim or in which situations it might be an alternative.

Virtual Private Servers

VPS is a technology that was developed for cheap web hosting by being able to share expensive hardware resources by as many web servers are possible. Compared to basic web hosting services, it additionally allows the web site provider to use some advanced web hosting features, like server side scripts or to use of database storage. This are useful features for better looking and more feature rich web sites.

VPS bases on the assumption that web sites of smaller and medium sized institutions usually do not get traffic most of the time. During these times the processes of such web sites can be put asleep and the data is temporarily
offloaded on slow hard drives to free main memory for currently active web sites. This allows many web sites to share the same physical resources (CPU, memory, bandwidth), which keeps costs low.

VPS providers apply a so called Fair Use Principle. If one hosting customer uses too many ressources (CPU, memory or bandwidth) they take measures to ensure that the overuse of one customer does not affect the service quality of other customers. This can mean that the available bandwidth of that customer is cut back or that the processing power is reduced to ensure nobody else is affected negatively by that overuse.

Now the question is, if VPS as suitable for OpenSim or in which cases VPS might be an alternative to host OpenSim.

OpenSim processes, like processes of other real time applications like voice-over-ip or video conferencing never sleep like web server processes. Even if there is no visitor in an OpenSim region, a typical OpenSim process anyway constantly consumes between 0.5 and 3% processing power of a processor core.

This permanent use of processing power is untypical for web server applications VPS was mainly developed for. Depending on the kind of visualization software used, such processes are anyway put asleep from time to time, to avoid negative impact on other VPS instances, although these processes should be kept running. This is done to enforce the Fair Use Principly and to ensure that other VPS instances on the same hardware server get enough computing power and physical main memory.

As a consequence OpenSim users experience lag, because of regular storing and loading of data to slow hard drives and because of the overhead and delays of constantly having to start and stop processes.

These negative effects increases the more users are on one hardware server providing VPS services. And it gets even worse if there are more such processes overusing resources on the same server hardware. Such processes with permanent resource demands begin to fight for the sparse hardware resources, because the basic assumption that most (web server) processes can sleep and offload data most of the time is not true anymore.

Bandwidth can be another issue, because OpenSim creates peaks of data traffic. Depending on the virtualization technology used, this also causes issues, if the peak bandwidth is limited. That can cause crashes and instabilities during logins and teleports.

Because VPS tries to store data on hard drives as often as possible to safe main memory, the additional time needed to load data from that slow storage cause additional lag. The probability of such delays increases the more memory an OpenSim process uses. Bigger regions with many prims, scripts and visitors quickly become unusable because of the permanent swapping or paging of memory needed by these OpenSim process.

This effect also increases the probability that Mono threads block or crash completely, if OpenSim runs on top of Mono. This causes instabilities and more frequent crashes of OpenSim regions.

What a VPS Hosting Company Says

I had a long discussion with the CTO of a big VPS hosting company and he did clearly say: “VPS is not suitable for any real time application [voice, video streaming or virtual worlds].”

He said once they did host a voice conference on a VPS server for a customer, but that this was just done for a marketing show. In reality that customer was the only one on the hardware server used, to ensure high service quality. So in fact that customer did use a dedicated server for that marketing campaign, although it was said that it’s a virtual server.

Summary

VPS is not suitable for regions with much contents (prims and scripts) or if many visitors need to be supported. Beside that the service quality is substantially lower: Visitors experience more lag, less stability and more frequent region crashes.

On the other hand, if someone just needs a small region with not too many prims and scripts and if just 2 or maximum 3 visitors are enough, then VPS offers a cheap alternative for beginners. But most region owners will quickly reach the limit and will want to be able to add more contents to their region, without having to experience a quickly degrading service quality.

For professional users, like corporations, shop owners, dance club owners, etc. VPS is definitive no alternative for professional OpenSim hosting using dedicated server hardware.

Dedicated Servers

Dedicated servers are the best choice for bigger regions and regions that have to support many visitors.

The prices of OpenSim regions hosted on dedicated servers very much depend on the processing power, main memory and network bandwidth provided. Good offers for beginners start with 512 MB main memory per region, while the typical average users need up to 1 GB. High end users that use many prims, scripts and have many visitors need up to 2 GB of memory for one region.

General Advices

In any case be careful if an OpenSim hosting providers promises you very high numbers of prims, scripts and visitors, especially if no information about the underlying hardware platform is given or if you see that the amount of memory, the included network bandwidth or processor seem not to be sufficient for what they promise.

At the end it only counts what you get and not what they did promise you, despite how low their price is!

You should also carefully check what is included in the hosting service package. Do they ensure high availability with professional 7×24 service monitoring? What kind of web based or in-world management tools are provided? What about backups? Are optional features like voice, search, groups, off-line messages, money, HyperGrid and mega regions included?

Data backups are a must and should include daily backups and on demand backups to protect you from data loss. Database backups are the most important, because they store all important data, but customers should also be able to get backups of their region contents as OAR archive files.

Customers should also get web based and/or in-world tools to manage their regions (display status, restart, etc.) and in-world monitoring panels for regions.

Finally and most important check if they have good references for their hosting services and for customer service? Service quality very much depends on how professionally OpenSim is managed and run. Good references are an indication if a hosting provider does his job well.

Currently many OpenSim hosting companies out there have problems to deliver what the promise. With good references the chances are higher that you will be happy customer, using good OpenSim hosting services for a reasonable price.

Snoopy Pfeffer

CEO & Founder of Dreamland Metaverse
snoopy.pfeffer@yahoo.com
http://www.dreamlandmetaverse.com/

“Life 2.0″ Documentary

PalmStar Entertainment and Andrew Lauren Productions did announce that “Life 2.0” (life2movie.com), directed by Jason Spingarn-Koff, has been accepted into the 2010 Sundance Film Festival. This film is a documentary about the impact of virtual communities on the world.

While most of the publications about virtual worlds focus on technology and business, this documentary puts a spotlight on the social and personal impacts virtual world communities can have on people.

“This feature length documentary follows a group of people whose lives are dramatically transformed by a virtual world, Second Life – reshaping relationships, identities, and ultimately the very notion of reality.”

Movie trailer:
http://www.life2movie.com/

Further information:
http://www.palmstar.com/2009/12/03/life-20-accepted-into-sundance-film-festival-2010/

Film Credits:
Presented by Andrew Lauren Productions and PalmStar Entertainment
Produced, Directed, and Edited by Jason Spingarn-Koff
Producers: Andrew Lauren and Stephan Paternot
Co-Producer: Jonathan Shukat
Director of Photography and Consulting Producer: Dan Krauss
Music by Justin Melland
Additional Editor: Shannon Kennedy

PayPal Money Module



After Adam Frisby did announce his DTL PayPal Money Module for OpenSim about a month ago, which got much positive feedback, I did test it intensively and I did extend it’s functionality. Last week Adam and I did decide, that it is the best if I continue to work on it using a fork of his GIT repository.

You can find that GIT repository with the latest version of the PayPal money module for OpenSim here:  http://github.com/SnoopyPfeffer/Mod-PayPal This fork has been renamed from DTL-PayPal to Mod-PayPal to avoid confusion.

Functional Extensions and Bug Fixes

  • User2User pay, User2Object pay, User2Object purchases and User2Land purchases work properly, now
  • User2User payments are confirmed with instant messages to senders and receivers
  • Support for group owned objects and land can be enabled (default: off); requirement: PayPal email accounts have to be defined per group
  • Underscores in email addresses do not cause error messages during OpenSim startup anymore
  • Object and land purchases for US$ 0 do not initiate a PayPal transaction anymore
  • Object purchases properly transfer the purchased contents to the buyer, now
  • User email addresses are loaded from the local OpenSim.ini file; invalid email adresses are ignored, but loading is not aborted anymore
  • User email addresses can also be fetched from the Users grid service, if that feature is enabled (default: off); fetched email addresses are cached by the region server until region restart; local email addresses always take precedence, to ensure an acceptable level of security
  • Locally defined user email addresses should be used for all users that receive bigger amounts of money within a region (i.e. shop owners); the reason is the much higher security, if these email addresses are not fetched from the Users grid service; beside that it is possible to locally use a special PalPal micropayment account for all shop transactions to save fees, instead of a standard PayPal account used elsewhere
  • Group email addresses can be defined per group UUID in the local ini file to support group owned objects and land as mentioned before
  • Non existent user or group email addresses are handled by showing warning messages, instead of error messages for crashed code
  • Added start and success log messages for all kinds of PayPal transactions
  • Rounding problems made it impossible to pay certain amounts (like US$ 1.23); now all amounts work and are displayed nicely

Interesting findings

  • PayPal can send money to all email accounts, even to email accounts not registered at PayPal yet; thus it is not necessary to handle that case
  • PayPal micropayment accounts are not available in all countries worldwide

To Do List

  • Web pages that are shown after successful or cancelled PayPal transactions
  • Locking for PayPal transactions while buying land or original objects
  • Events or functions that allow to lock vendors while a PayPal transaction is in progress
  • Too small amounts that do not cover the PayPal transaction fees produce a misleading error message on the PayPal web page; maybe it would be good to be able to define a minimum amount globally and/or per email address to avoid that
  • It would be good if a scripter using llGiveMoney gets a warning message, that this function is not supported by the PayPal money module
  • User2User payments are confirmed with instant messages to the sender and receiver of money, but only if these users are online at that time; maybe it is possible to improve that, so that even users offline get such confirmation messages as stored offline messages

Installation of the PayPal Money Module

  • Install the addon-modules/mod-paypal under OpenSim/Region/OptionalModules/
  • Install the files-for-bin-dir in the OpenSim bin folder
  • Add the additional config settings in the config-include subfolder or in the OpenSim.ini file in your bin folder
  • After that compile OpenSim

If you find bugs or if you have ideas for the future development of this module, please send me an email (snoopy.pfeffer@yahoo.com). Thank!

Hosted Regions in OSGrid

My company Dreamland Metaverse offers high quality regions in OSGrid and standalone regions based on open source OpenSim virtual world servers. Our goal is to provide our renters a good service for a fair price.

Since over 1.5 years our regions in OSGrid are known for their high reliability, using the latest stable OpenSim versions. Our good references show, that we are able to maintain high service quality.

Many residents have chosen our regions (like the shopping mall regions Samsara and Snoopies) as their login locations, because our regions run more reliably than the main OSGrid plazas.

Because we focus on quality, we have chosen the following server configuration:

  • We focus on best quality regions for OSGrid and as standalone regions
  • Dedicated, high performance Intel Quad-Core Xeon servers optimized for OpenSim
  • Enough computing power, memory and bandwidth for a low lag experience
  • Servers located close to the central OSGrid servers for higher performance
  • Service monitoring tools for 7×24 availability
  • Daily data backups and redundant RAID 1 hard drives
  • Import and export of region contents (OAR archives)
  • All region owners get estate manager rights
  • Land can be split, joined, subleased and sold
  • Web portal with service management tool for our customers
  • Search, groups, offline messages, voice and money (PayPal), HyperGrid

We offer the following region types, where the minimum renting duration is 3 months.

  • Basic – OSGrid or Standalone Region
    Regions mainly intended for landscapes and regions with low building density

    8’000 prims, 10 visitors, shared Intel quadcore, 512 MB memory
    no setup costs; monthly rent: US$ 30
  • Residential – OSGrid or Standalone Region
    Best solution for residential land – with the capacity you need for a typical region
    12’000 prims, 40 visitors, shared Intel quadcore, 1 GB memory
    no setup costs; monthly rent: US$ 45
  • Professional – OSGrid or Standalone Region
    Regions for business and event areas – allows many prims, scripts and visitors
    16’000 prims, 80+ visitors, dedicated Intel core, 2 GB memory
    no setup costs; monthly rent: US$ 90

If you are interested, you can order a region on our web site. I am happy to answer any questions.

Dreamland Metaverse
Region Hosting & Consulting

Snoopy Pfeffer
snoopy.pfeffer@yahoo.com
www.dreamlandmetaverse.com

The Costs of Free Resources

We all like it: When we get something for free!

But the non existence of costs has also disadvantages: We tend to ignore the resources that are associated with what we use. This is simply the case, because there are no costs for us, that otherwise would provide an incentive to make a more economical use of resources.

In OSGrid users like that it is possible to upload textures, animations and sound files for free. Additionally the users do not have to pay for activities like creating new groups. To make it clear: I fully back these policies, because they support the development of OSGrid.

And at this point we should all  thank the donators that cover the costs of the central infrastructure of OSGrid and that help keep it running so well every day. For sure you all know about the new “donate” button on the new Elgg based OSGrid web site.

But the missing incentives to save resources has negative effects, that over time increase the costs for central grid services unnecessarily, reduce the performance and lessen usability in certain areas.

In the following paragraphs I pick two examples to describe these issues and how these could be solved by implementing policies and tools. With this article I want to initiate a discussion about these issues, before we experience a future rapid grows of free Opensim grids.

A)  Asset Usage

People tend to upload textures, animations and sound files even if they exist within the world already. From the user point of view the costs – the time needed to check if for example a texture is available already – is higher than simply uploading it once more. Such duplicates inflate the space required for the asset database and asset caches. Additionally duplicates increase the data that needs to be transferred and thus reduces Opensim performance.

Additionally people tend to upload multiple versions of an asset to test it in world until they are happy with the result. At the end they just use the last version. Such unused assets, assets that are not referenced anymore, still consume space in the asset database that could be freed.

Possible Solution

One solution would be to introduce incentives for a more economical use of these resources. Second Life does this with their L$10 uploading fee.

But even better would be, if Opensim is able find duplicates and unused assets. Alternatively this can be done on the fly, by using some Opensim extension, or by using database scripts that are run regularly, as part of an asset database maintenance task. The goal is to remove all unreferenced assets and to replace duplicate assets with a single copy, which might include the replacement of the corresponding UUIDs.

To efficiently find duplicates, data block sizes and hash values can be used. For sure only assets that cannot be changed by individual users separately can be replaced by a single asset copy. This is especially true for uploaded textures, animations and sound files, that cannot be changed within the viewer. Notecards and scripts, that can be changed in world, and other changeable assets cannot be replaced by a single asset database copy.

In practice OSGrid uses Fragstore to do dupe compression based on 256 bit hash values for nearly 3 million assets, now. The compression achieved is about 25 percent. Currently no deletion of unused assets is done. The detection of unused assets seems to be impossible or at least very difficult with the current asset server architecture Opensim uses.

B) Creating New Groups

Groups have been introduced with Opensim revision 9215, but already today there exist nearly 600 groups in OSGrid. I have had a closer look at the existing groups and saw the following:

  • Most groups just have one member
  • There are even many groups with no members
  • Some groups have very similar names and it looks like that most of them have already been given up for newer groups with a slightly improved title

If the development continues like this, we soon will see many thousand groups, where it is difficult to identify which groups are trash and which are still used. For certain keywords group search already returns many groups with many similar names (search for “star” for example).

Possible Solution

To prevent an exaggerate use  of groups, that also reduces usability, policies and tools that support these policies should be used.

In Second Life people have to pay a fee to create a new group. Additionally the group owner has to recruit more group members within 3 days, otherwise the group gets deleted. And a group must maintain a membership of at least two members all the time in order to remain active.

In OSGrid I think we do not need a fee, but a policy for active groups similar like in Second Life. This means that empty groups and groups with less than 2 members would get deleted at latest 3 days after creation. This is the case for most groups that currently exist in OSGrid.

Additionally it might be useful to check when all group members have logged in the last time. If no group member has logged in during the last 90 days, it can be assumed, that all group members are not active OSGrid residents anymore.

A database script could check all existing groups regularly and delete groups that do not fulfill the previously mentioned conditions. For sure Opensim needs to behaves properly, if a group is encountered that was deleted and if such a group is used by land or objects shared by or deeded to that group.

In the case of shared objects and land, the group defined should simply be removed, making it unshared objects or land again. Deeded objects and land is more tricky, because in that case the group also needs to be deleted and the owner has to be reset to the previous owner before the object or land was deeded. All this should just happen if that group really was deleted. A disabled group module (maybe by accident) should not reset all group memberships.

Well, the last part was a bit technical, but it shows that implementing such a policy is something that needs to be well thought through.

Finally a little history story: As we all know, teleporting between regions also uses resources, especially processing power and bandwidth to transfer data. That is why during Second Life “stone age” they tried to introduce a fee for teleporting between regions. But understandably it did not take long, until users were demanding to abolish this teleporting fee.

I am sure that we will find a solution to prevent an exaggerate use ressources with policies and tools, without having to use money as incentive.

Group Permissions for Land and Objects

A while ago, mcortez and his flotsam project have provided Opensim the source code to add group functionality to Opensim. And after a short interruption, because of security issues that needed to get fixed, it is now part of all newer Opensim versions and available at most OSGrid regions.

The Groups Module allows people to define and join groups. So today we all can use fancy group tags like in Second Life, which are defined as part of the group roles. In addition it is possible to send and receive group instance messages and group notices. Voice conferences can be used if all attendees are on voice enabled regions using the same voice server.

What was still missing were group permissions for land and objects. These allow land and object owners to define the rights other people have to change the contents of their parcels of land or to change objects itself.

This functionality is essential for collaborative building projects, where you want to allow just a specified group of builders to be able to rez or change objects. Till now, you had to allow all people to build on your land and to modify common objects (by tagging “allow anyone to move”). This was quite risky, because griefers could rez objects or even destroy what you have built together.

With today’s Opensim trunk version 9817 group permissions for land and objects have been added to Opensim.

To get group permissions working, I had to add group permission checks to the Opensim Permissions Module. This new functionality is very similar like in Second Life, but currently there are still limitations, because some related functionality has not been implemented in Opensim yet.

For example it is not possible to define object inventory items as being shared. And currently it is not possible to allow group members to edit certain settings of land they do not own in the About Land window. Thus currently it is not wise to deed land to a group, but land shared by a group, with the corresponding rights set in About Land, is a very useful new feature.


Blog Stats

  • 17,211 hits

Pages

April 2014
M T W T F S S
« Jun    
 123456
78910111213
14151617181920
21222324252627
282930  

Follow

Get every new post delivered to your Inbox.

Join 133 other followers